Brian and I have plotted the beginning of the (bad) excitation in dataviewer in both the ISI and IOP models. We have also plotted the GPS time according to the ISI model (DCU_ID 75). As you can see, the GPS time drops out when the model is being off, but comes back up shortly thereafter. This data is for the 11:42 UTC restart when the excitation starts. 

We plot the same for the second restart, twenty minutes after which the fiber break occurs. We see the same thing - the model is shut down, then starts again. We see a similar drop in the GPS time to 0 before the model restarts. The excitations also stop when the model stops, and start again when the model restarts. We see a growth in the amplitude of the coil drive signal as well as the DAC signal from the IOP.

This is not inconsistent with a loop instability within the model.

Yesterday, we measured the resistances of the 12 actuators on ISI-BSC8. The table below shows the measured resistance when the ISI was:

- in the staging building

- on the teststand in the LVEA

- in the chamber (March 6, 2012)

Field cables in the LVEA are about 1 ohm. Resistances measured after the important drive are similar to those measured when the ISI was on the LVEA teststand. There are no obvious signs of damage on these actuators.

Betsy B. Travis S. Thomas V.

After reinstalling the swing back plate and support brackets, we swung the ACB back so that SUS can access the quad somewhat freely.

200W beam

35W beam

The plots in the document attached show calibrated motion and velocity of the ISI when it reaches the steady state regime that will last for hours and lead to the fiber braking.

Ed (Apollo), Michael R

Ducting for the exhaust air path was installed yesterday. Installation is not complete, as we are waiting on some parts, and I still need to caulk the penetration into the enclosure.

After swapping the picomotor cable, I and Dan Hoak proceeded to check the grounding. To our disappointment, the ISC table is grounded to three of the four cable shells.

We disconnected all four cables connecting the ISC table to the bracket on the ISI table, and tested the grounding of the ISC table to the on-ISC-table cable shells (there are four), and they're fine.

Somewhere downstream, shielding of one or more cables should be touching the metal part of the TMS or ISI, and of course the ISC table is electrically grounded to any of these structures. We need to check cable clamps and copper wires used for bundling the cables.

• Originally pin1, 6, 7 and 13 were all connected (in-vac side) on Mar 02.
• It was divided into two groups, 1-13 and 6-7 when I wiggled the cable on the cable bracket on ISI on Mar 02.
• Then somehow pin 6 and 7 were disconnected from each other when I disconnected and reconnected the cables on ISI on Mar 02.
• Through this process and other measurements, we identified that pin1-13 problem was in D1000223-S1104079, but could not identify the source of pin6-7.
• On Mar 06 morning, before the cable was swapped, we repeated the test. We've found that pin6 and 7 were again connected. Something apparently crept  back, though we don't know what.
• After swapping the cable, everything was fine on Mar 06.
• There's still some chance that pin6-7 will creep back to bad status (if the problem was not in the swapped cable). Pin 6 and 7 are used for red QPD picomotors, which are not used for one arm test.

Late entry from Monday morning.

We had a network disconnect from the LX vacuum controls system. This was tracked to a bad ethernet cable in the LVEA and it was replaced. The data gap was of the order of an hour. Only h0velx was affected.

At this mornings meeting, it was decided that Keita would add the locking mechanics to the TMS in order to lock it and continue investigating cable and grounding issues.  Once the hardware issues are settled, the locking hardware will be removed and SUS/SEI testing can commence.

A detailed description of the CDS front ends controlling BSC8 at the time of the fiber breakage has been submitted to DCC as document: LIGO-T1200123

https://dcc.ligo.org/cgi-bin/private/DocDB/ShowDocument?.submit=Number&docid=T1200123&version=

there is sufficient detail in this report that I did not want to submit a huge alog entry. Later I'll also submit a tar file of the front end models and related files covering this time period.

(Bland, Sadecki, Shankle, Jodi, Apollo-ers)

This morning, the BSC8 South door was pulled.  Taking care to set down foil stepping stones on the rubble laden flooring in the chamber, I worked my way towards the ITMy shooting pictures along the way.  Travis also entered the chamber for inspection.  We've started 3 libraries of pictures on ResourceSpace, detailed below.

Floor debris from the glass fibers:
https://ligoimages.mit.edu/?c=992

General chamber pictures of ACB and ITMy:
https://ligoimages.mit.edu/?c=991

Some metal shavings were found on the top of the ACB - likely shaken loose from the rogue EXC, but left behind from the Stage 0 hole tapping which occurred during the initial ACB installation.  More notes on this in a subsequent alog.  These metal shavings may or may not be the cause of the fiber breakage, although they are in the vicinity of the fibers.  We hope to do a more thorough search tomorrow for more metal flakes on the ITMy suspension.  At first pass, the 14 collected metal bits were found on the ACB top surface, while none were observed on the flooring of the chamber.

ITMy ears and horns:
https://ligoimages.mit.edu/?c=990

2 of the 8 horns at the base of the fibers have broken off a bit low.  SUS is evaluating rewelding on these horns.

After inspection the following steps were performed:
- With tweezers, we picked up the largest bits of glass stock and fiber and collected them in petri dishes while searching for other non-glass debris.  Only one fleck of metal and a few cloth type fibers were found besides the glass debris.  The one fleck of metal was found many feet from the ITMy, back and to the NE of the FMy SUS.
- The floor was wiped from back to front, pulling most of the glass particulate out towards the door.  Wet wipes were used.  More wiping will be needed.
- The ISI was locked down by Jim, from underneath the ISI table (standing on a stool on flooring in chamber).
- The BSC repair arm and all of it's associated hardware was mounted to the door flange.

The attached plots are detailed data trends for all input and output channels on the H2 SUS ITMY M0/R0 top masses and each stage of the BSC-ISI during a 50-second period last Thursday March 1 beginning at approximately 11:50:00 PST (UTC: 19:50:00 , GPS: 101 466 6670) until the trigger of the SUS ITMY software watchdogs. 

Some notes before diving into these trends:
1.) The first trigger of the SUS ITMY WatchDogs was at approximately:
    101 466 6686 GPS
    19:51:11     UTC
    11:51:11     PST
2.) These plots all contain 50 seconds of full frame data for those channels with data being written to frames.  Each attachment has a plot of the three SUS ITMY WatchDog states when they first tripped at approximately (GPS) 101 466 6686.

3.) The FEC channels are the EPICS data from the DAC outputs for either the SUS or SEI machines (h2susb478 and h2seib478), which is sampled at 16Hz.  These signals should be the outputs sent to coil drivers.

4.) Each attachment is a different set of either inputs or outputs for each sensor/actuator group on either the BSC-ISI or the ITMY M0/R0 top mass coils. This is true for about the first six plots on each attachment - the SUS WDs and M0 COILOUTF channels were kept while trending different sensor/actuator groups.  

-----------------------------------------------------------
Attachments:
1.)  M0 OSEMINF, M0 WatchDog states, M0 COILOUTF (DAQ data written from output of SUS coil drivers)

2.)  M0 OSEMINF, M0 WatchDog states, M0 DAMP OUTMON (epics 16Hz data)

3.)  M0 OSEMINF, M0 WatchDog states, FEC-30 DAC OUTPUT (DAC machine generating the ITMY coil output signals)

4.) R0 OSEMINF, M0 WatchDogs, R0 COILOUTF

5.) ISI ST1 CPS inputs

6.) ISI ST1 L4C inputs

7.)ISI ST1 T240 inputs

8.) ISI ST2 CPS inputs

9.) ISI ST2 GS13 inputs

10.) ISI FEC-75 (h2seib478 DAC output channels - first half of channels)

11.) ISI FEC-75 (h2seib478 DAC output channels - second half of channels)

--------------------------------------------------------

From these trends, one can deduce several points about which system was actuating just before the break of the ITMY fibers:
1.) The SUS ITMY M0 and R0 damping filters were CLOSED and damping the top masses during the moments leading to the break.

2.) There was excessive motion seen on ALL SUS ITMY M0/R0 OSEMS and each sensor group of both stages of the BSC-ISI.

3.) The SUS ITMY M0 Watchdogs tripped and cut off the output of the DAC signal from h2susb478.  

4.) The ISI DAC machine (h2seib478) was outputting signals on at least 6 of it's outputs.  The exact corresponding ISI channels are not apparent, but the DAC FEC data shows which specific DAC channel had a signal. This indicates the exact actuators with input signals cannot immediately be determined, but not impossible.   

5.)  The ISI Watchdogs were triggered and turned off the "*_MASTER_SWITCH_*".  However, a residual signal was present on the ISI's DAC outputs after the WD tripped.

Pods #94 and #71 were experiencing the problems laid out in alogs 2308 and 2321. So today I de-podded both of them. During the de-podding I noticed no issues with the feet or any other kinematic devices.
I did find a broken flexure on #71, however (pictures attached). Which is odd since the GS-13 tested fine on arrival. Somewhere in between its arrival and testing it completely failed. It's possible it was severally fatigued during shipping and only recently failed.
Pod #94 has had its brains swapped with another failed GS-13 (mechanical issues) and will hopefully be tested sometime tomorrow to see if that solves the gain issues. If not that should point us toward some mechanical problem that I haven't seen before.
Both of these pods will become test pods (if their issues can be fixed) until they can be sent to LLO for retesting.

        In cdsimac6.cds.ligo-wa.caltech.edu:/Users/controls/2012_03_06_channelStudy, I left my notes from a study to see which of the ~250-270 excitations could have gone wrong. I first exported all of the channels on that ISI with "EXCMON" in the name to channelList.txt. I then copied that file to channelListDone.txt.
	Within channelListDone.txt (attached), I marked a plus by a channel that had abnormal activity between 0800 and 1600 PST on 3/1/2012. A minus is marked by a channel that I checked that seems clear. I've checked 144 channels by running each of them in Data Viewer. I've also attached the latest version to this post.
	See attached PDF for a sample Data Viewer plot of five suspect channels and one innocent channel (complete flatline). I don't notice any patterns so far to which channels were triggering. However, I hope to look at the other ~150 channels soon.

- Investigation into rogue excitation in BSC8 ISI, possibly causing fiber to break. Door will come off.
- BSC6 - looking at south collimator, because of high beam divergence.
- BSC6 - After IAS work, swap cable that connects feedthrough and ISI table for TMS picomotors. 
- Apollo is working on mezzanine electronics
- Bob Rhodes working on HVAC with man-lift in OSB receiving, H1 electronics area, drilling for decking
- Apollo working on split units in MSR
- H1 PSL installing fan panels
- Portable bathroom servicing is here, King Soft as well.
- Paradise Water delivery at 0909 PST
- UniFirst at ~0915
- Praxair at 0927
- Filiberto and Keita at EY to test cables
- 1018 - Bob Rhodes Heating and Air arrives
- 1516 - Control Solutions Northwest arrives for Ski
- 1600 - OSB doors locked

The BSC8 HEPI actuators were placed in run configuration with fluid flowing through them.  Immediately after the fluid was valved to bypass the actuators at 4pm Monday 3/5/12 by turning the 4-way bypass valves into recirculation mode.  The pressure in the line out to the chamber is set at 70psi.  

Today SUS testers noted ITMY M0 top OSEM sensors railed (see Kissel/Barton alog entries below, 07:47 today).  Concerned for the state of the ITMY monolithic fibers, we pulled viewport covers to inspect the quad.  While the view of the TM was largely occulted by the AOS ACB, we could see evidence of fiber breakage on the floor in the form of glass shards.  The ISI/SUS/HEPI underwent a major and sustained actuation.  Investigations are underway to determine the cause.  

We will open BSC8 early next week to begin repair work.

-Landry/Bland/Sadecki

Fabrice, Hugo,

A GS13 huddle test was performed in order to pursue the investigation of the gain difference observed on the TFs measured on HAM-ISI Unit#2 (alog #2308)

Three horizontal GS13s were tested on the “bench”

• Pod #94, Production

Initially installed on this unit. Gain is half too small. One of its two op-amp could be malfunctioning.

• Pod #71, Production Spare

Response lower by a factor of 100 to 10000 below 20Hz. The seismic mass can be felt/heard when tilting the instrument, hence it is not stuck. The instrument will be opened at LHO for inspection. It will have to go back to LLO to pass the leak-checking-test before being installed on a Unit.   

• Pod #66: Test instrument

Locked test-mass got unlocked after manipulation. Acceptable power-spectrum. Not vacuum-compatible.

Conclusion:

Production GS13s are malfunctioning. The Test GS13 Pod #66 will be used temporarily for this testing phase of HAM-ISI Unit #2 testing. It will be removed before storage of the unit. Later on, a production GS13 will be installed for the side-chamber testing.