At this mornings meeting, it was decided that Keita would add the locking mechanics to the TMS in order to lock it and continue investigating cable and grounding issues.  Once the hardware issues are settled, the locking hardware will be removed and SUS/SEI testing can commence.

A detailed description of the CDS front ends controlling BSC8 at the time of the fiber breakage has been submitted to DCC as document: LIGO-T1200123

https://dcc.ligo.org/cgi-bin/private/DocDB/ShowDocument?.submit=Number&docid=T1200123&version=

there is sufficient detail in this report that I did not want to submit a huge alog entry. Later I'll also submit a tar file of the front end models and related files covering this time period.

(Bland, Sadecki, Shankle, Jodi, Apollo-ers)

This morning, the BSC8 South door was pulled.  Taking care to set down foil stepping stones on the rubble laden flooring in the chamber, I worked my way towards the ITMy shooting pictures along the way.  Travis also entered the chamber for inspection.  We've started 3 libraries of pictures on ResourceSpace, detailed below.

Floor debris from the glass fibers:
https://ligoimages.mit.edu/?c=992

General chamber pictures of ACB and ITMy:
https://ligoimages.mit.edu/?c=991

Some metal shavings were found on the top of the ACB - likely shaken loose from the rogue EXC, but left behind from the Stage 0 hole tapping which occurred during the initial ACB installation.  More notes on this in a subsequent alog.  These metal shavings may or may not be the cause of the fiber breakage, although they are in the vicinity of the fibers.  We hope to do a more thorough search tomorrow for more metal flakes on the ITMy suspension.  At first pass, the 14 collected metal bits were found on the ACB top surface, while none were observed on the flooring of the chamber.

ITMy ears and horns:
https://ligoimages.mit.edu/?c=990

2 of the 8 horns at the base of the fibers have broken off a bit low.  SUS is evaluating rewelding on these horns.

After inspection the following steps were performed:
- With tweezers, we picked up the largest bits of glass stock and fiber and collected them in petri dishes while searching for other non-glass debris.  Only one fleck of metal and a few cloth type fibers were found besides the glass debris.  The one fleck of metal was found many feet from the ITMy, back and to the NE of the FMy SUS.
- The floor was wiped from back to front, pulling most of the glass particulate out towards the door.  Wet wipes were used.  More wiping will be needed.
- The ISI was locked down by Jim, from underneath the ISI table (standing on a stool on flooring in chamber).
- The BSC repair arm and all of it's associated hardware was mounted to the door flange.

The attached plots are detailed data trends for all input and output channels on the H2 SUS ITMY M0/R0 top masses and each stage of the BSC-ISI during a 50-second period last Thursday March 1 beginning at approximately 11:50:00 PST (UTC: 19:50:00 , GPS: 101 466 6670) until the trigger of the SUS ITMY software watchdogs. 

Some notes before diving into these trends:
1.) The first trigger of the SUS ITMY WatchDogs was at approximately:
    101 466 6686 GPS
    19:51:11     UTC
    11:51:11     PST
2.) These plots all contain 50 seconds of full frame data for those channels with data being written to frames.  Each attachment has a plot of the three SUS ITMY WatchDog states when they first tripped at approximately (GPS) 101 466 6686.

3.) The FEC channels are the EPICS data from the DAC outputs for either the SUS or SEI machines (h2susb478 and h2seib478), which is sampled at 16Hz.  These signals should be the outputs sent to coil drivers.

4.) Each attachment is a different set of either inputs or outputs for each sensor/actuator group on either the BSC-ISI or the ITMY M0/R0 top mass coils. This is true for about the first six plots on each attachment - the SUS WDs and M0 COILOUTF channels were kept while trending different sensor/actuator groups.  

-----------------------------------------------------------
Attachments:
1.)  M0 OSEMINF, M0 WatchDog states, M0 COILOUTF (DAQ data written from output of SUS coil drivers)

2.)  M0 OSEMINF, M0 WatchDog states, M0 DAMP OUTMON (epics 16Hz data)

3.)  M0 OSEMINF, M0 WatchDog states, FEC-30 DAC OUTPUT (DAC machine generating the ITMY coil output signals)

4.) R0 OSEMINF, M0 WatchDogs, R0 COILOUTF

5.) ISI ST1 CPS inputs

6.) ISI ST1 L4C inputs

7.)ISI ST1 T240 inputs

8.) ISI ST2 CPS inputs

9.) ISI ST2 GS13 inputs

10.) ISI FEC-75 (h2seib478 DAC output channels - first half of channels)

11.) ISI FEC-75 (h2seib478 DAC output channels - second half of channels)

--------------------------------------------------------

From these trends, one can deduce several points about which system was actuating just before the break of the ITMY fibers:
1.) The SUS ITMY M0 and R0 damping filters were CLOSED and damping the top masses during the moments leading to the break.

2.) There was excessive motion seen on ALL SUS ITMY M0/R0 OSEMS and each sensor group of both stages of the BSC-ISI.

3.) The SUS ITMY M0 Watchdogs tripped and cut off the output of the DAC signal from h2susb478.  

4.) The ISI DAC machine (h2seib478) was outputting signals on at least 6 of it's outputs.  The exact corresponding ISI channels are not apparent, but the DAC FEC data shows which specific DAC channel had a signal. This indicates the exact actuators with input signals cannot immediately be determined, but not impossible.   

5.)  The ISI Watchdogs were triggered and turned off the "*_MASTER_SWITCH_*".  However, a residual signal was present on the ISI's DAC outputs after the WD tripped.

Pods #94 and #71 were experiencing the problems laid out in alogs 2308 and 2321. So today I de-podded both of them. During the de-podding I noticed no issues with the feet or any other kinematic devices.
I did find a broken flexure on #71, however (pictures attached). Which is odd since the GS-13 tested fine on arrival. Somewhere in between its arrival and testing it completely failed. It's possible it was severally fatigued during shipping and only recently failed.
Pod #94 has had its brains swapped with another failed GS-13 (mechanical issues) and will hopefully be tested sometime tomorrow to see if that solves the gain issues. If not that should point us toward some mechanical problem that I haven't seen before.
Both of these pods will become test pods (if their issues can be fixed) until they can be sent to LLO for retesting.

        In cdsimac6.cds.ligo-wa.caltech.edu:/Users/controls/2012_03_06_channelStudy, I left my notes from a study to see which of the ~250-270 excitations could have gone wrong. I first exported all of the channels on that ISI with "EXCMON" in the name to channelList.txt. I then copied that file to channelListDone.txt.
	Within channelListDone.txt (attached), I marked a plus by a channel that had abnormal activity between 0800 and 1600 PST on 3/1/2012. A minus is marked by a channel that I checked that seems clear. I've checked 144 channels by running each of them in Data Viewer. I've also attached the latest version to this post.
	See attached PDF for a sample Data Viewer plot of five suspect channels and one innocent channel (complete flatline). I don't notice any patterns so far to which channels were triggering. However, I hope to look at the other ~150 channels soon.

- Investigation into rogue excitation in BSC8 ISI, possibly causing fiber to break. Door will come off.
- BSC6 - looking at south collimator, because of high beam divergence.
- BSC6 - After IAS work, swap cable that connects feedthrough and ISI table for TMS picomotors. 
- Apollo is working on mezzanine electronics
- Bob Rhodes working on HVAC with man-lift in OSB receiving, H1 electronics area, drilling for decking
- Apollo working on split units in MSR
- H1 PSL installing fan panels
- Portable bathroom servicing is here, King Soft as well.
- Paradise Water delivery at 0909 PST
- UniFirst at ~0915
- Praxair at 0927
- Filiberto and Keita at EY to test cables
- 1018 - Bob Rhodes Heating and Air arrives
- 1516 - Control Solutions Northwest arrives for Ski
- 1600 - OSB doors locked

The BSC8 HEPI actuators were placed in run configuration with fluid flowing through them.  Immediately after the fluid was valved to bypass the actuators at 4pm Monday 3/5/12 by turning the 4-way bypass valves into recirculation mode.  The pressure in the line out to the chamber is set at 70psi.  

Attached are plots of dust counts > .5 microns. I have included a plot of H0:PEM-LVEA_DST15_MODE to show when the dust monitor in the clean room over BSC8 (H0:PEM-LVEA_DST15_5) was disconnected.

The document attached shows trends from the ISI. The vertical red lines mark the approximate time of the failure. 

Page 1:

It shows the 12 actuators drive output of the BSC-ISI model. They were all zeros approximately 3 hours before and 3 hours after the fibers brake.


Page 2:

- It  shows the status of the ISI watch dog. It tripped approximately one hour before the brake.

- However, the comparison between the figure on the left and the figure on the right show a discrepancy between the "slow" and "daq" version of the same channel. According to the DAQ channel, the ISI front end was off when the fibers broke. According to the slow channel the watchdog was tripped. In both cases no excitation is sent from the ISI model to the DAQ, but we would like to understand why these information contradict each over.


Page 3:
- Channel 1 to 6 shows the Coil driver Voltage readback of stage 2. 
- Channel 7 to 12 shows the Coil driver current readback of stage 2.    
- Channel 13 shows that the ISI master switch was off
- Channel 14 shows that the ISI watchdog was tripped.

We wonder how the DAQ could drive the coil driver while the ISI real time controllers output were all zeros.


Page 4 shows the similar thing for Stage 1.

Here are some Dataviewer snapshoots of the event. Huge excitations start at 11:50am on March 1, 2012. (19:50 UTC)

Failure_3.png presents the coil drivers readbacks. Signals start going out of the coil drivers at 11:50am for no obvious reasons.

Models.png shows the last filter banks before the DAQ. The output channels are (:ISI-ST_MASTER_) and the 2 master switches are :ISI-ST_MASTER_SWITCHMON. All these signals were null at 19:50 UTC. This is not the case for the DAC channels (<IFO>-FEC_<DCU_ID>_DAC_OUTPUT_0_).

Failure_6.png shows the output channels of the ISI model and the DAC channels.

It seems that there are discrepancies between the ISI signals and the DAC signals.  

There were ETMY M0/R0 excitations beginning around 18:21:00 PST March 4th (2:21:00 UTC - March 5th) March 4th until the early hours of Monday March 5th (local time).  For quiet data with no actuation on the ETMY QUAD, any time before March 4 at 18:21:00 PST (02:21:00 UTC - March 5) up to the morning of Saturday March 3 (local PST time).

ISI Coil Driver electronics for both BSC6 and BSC8 are powered down.

Summary of changes made:

Last revision prior to changes is revision 1911. To reset Hanford userapps to the old ISI systems, please revert the HPI and ISI common and H2 folders to revision 1911 and re-make h2isiitmy and etmy, h2hpiitmy and etmy, . Revision 1912-1913 are changes to the master model including the new control. Revision 1913 and 1914 are medm screen revision, 1915 is controls on the master model. Revision 1916 includes the changes to h2isiitmy.

HEPI Changes:
On h2hpiitmy.mdl, I added in three ADCs: ADC0, ADC1, ADC2. These were all terminated. Prior to that, only ADC3 (card num = 3) was in the model. I also changed the bus outputs from the ADC selector, since they were previously connected with arrows copy-pasted from ADC0.
No changes were made to the hepi template.

ISI Changes:
For h2isiitmy, DAC_0 (card_num=0) and DAC_1 (card_num=1) were placed in ISI model - DAC_0 connected to ground entirely since it is shared with HPI. Previously, 

Lots of controls changes were made: feed forward and sensor correction from 0->1 and 1->2. These new controls all had the outputs from their output filters grounded so that no additional signal would be added to the model until 

Compiles - The models were compiled at various points to test out an ADC problem - the channels at the input to the ADC did not have the same value as the channels at the input to the HEPI model. At one point, the HEPI model was compiled at 4k to test.

In terms of what might have caused the motion craziness, the changes I made were either internal - prior to the overall output watchdog - or external - adding or changing ADC and DAC connections. The watchdog should have still prevented any model changes from driving the platform differently. The changes that might have caused problems with I/O inter-connect problems would have been 
1) adding the HEPI DAC to the ISI model, 
2) adding the three ADCs (0, 1, and 2) to the HEPI model, 
3) changing the connections from the HEPI ADC3 to the hepitemplate block, or 
4) recompiling the hepi model at 4k rather than 2k rate to test out the IOP problem we were having.

I'll edit/comment on this as more comes to mind, feel free to e-mail me additional questions or comments.

Still investigating, but wanted to break what appears to be the radio silence...

These are spectra of the HEPI sensors. Not sure if they're calibrated correctly, but *could* be in nm and nm/s.


This unphysically high Q in HEPI points to some digital problem

Best (ok, it's the current MIT guess, not necessarily the best) guess is some user model / IOP model exchange problem...

Today SUS testers noted ITMY M0 top OSEM sensors railed (see Kissel/Barton alog entries below, 07:47 today).  Concerned for the state of the ITMY monolithic fibers, we pulled viewport covers to inspect the quad.  While the view of the TM was largely occulted by the AOS ACB, we could see evidence of fiber breakage on the floor in the form of glass shards.  The ISI/SUS/HEPI underwent a major and sustained actuation.  Investigations are underway to determine the cause.  

We will open BSC8 early next week to begin repair work.

-Landry/Bland/Sadecki

Present status of ISC in-vac components on TMS are:

• QPDs work. This is important.
• Beam diverter itself works, but one of reed switches are stuck closed (ON). We will try to shim the sensor to push it far away from the magnet.
• One picomotor doesn't work (M1 Hor, this is the splitter for the green QPD and main path). This is likely the failure of the cable on the vacuum feedthrough (D1000223). Replacing this one is possible.

Fabrice, Hugo,

A GS13 huddle test was performed in order to pursue the investigation of the gain difference observed on the TFs measured on HAM-ISI Unit#2 (alog #2308)

Three horizontal GS13s were tested on the “bench”

• Pod #94, Production

Initially installed on this unit. Gain is half too small. One of its two op-amp could be malfunctioning.

• Pod #71, Production Spare

Response lower by a factor of 100 to 10000 below 20Hz. The seismic mass can be felt/heard when tilting the instrument, hence it is not stuck. The instrument will be opened at LHO for inspection. It will have to go back to LLO to pass the leak-checking-test before being installed on a Unit.   

• Pod #66: Test instrument

Locked test-mass got unlocked after manipulation. Acceptable power-spectrum. Not vacuum-compatible.

Conclusion:

Production GS13s are malfunctioning. The Test GS13 Pod #66 will be used temporarily for this testing phase of HAM-ISI Unit #2 testing. It will be removed before storage of the unit. Later on, a production GS13 will be installed for the side-chamber testing.

Due to high wind speeds, the South Bay roll-up door was closed to mitigate the elevated dust levels in the LVEA.  Normally, this door is kept open to purge air from the LVEA, but the increased wind speeds overcame the over-pressurised LVEA.  Dust monitor levels have subsided to normal levels after the close just before noon.