The BSC8 HEPI actuators were placed in run configuration with fluid flowing through them. Immediately after the fluid was valved to bypass the actuators at 4pm Monday 3/5/12 by turning the 4-way bypass valves into recirculation mode. The pressure in the line out to the chamber is set at 70psi.
Attached are plots of dust counts > .5 microns. I have included a plot of H0:PEM-LVEA_DST15_MODE to show when the dust monitor in the clean room over BSC8 (H0:PEM-LVEA_DST15_5) was disconnected.
The document attached shows trends from the ISI. The vertical red lines mark the approximate time of the failure. Page 1: It shows the 12 actuators drive output of the BSC-ISI model. They were all zeros approximately 3 hours before and 3 hours after the fibers brake. Page 2: - It shows the status of the ISI watch dog. It tripped approximately one hour before the brake. - However, the comparison between the figure on the left and the figure on the right show a discrepancy between the "slow" and "daq" version of the same channel. According to the DAQ channel, the ISI front end was off when the fibers broke. According to the slow channel the watchdog was tripped. In both cases no excitation is sent from the ISI model to the DAQ, but we would like to understand why these information contradict each over. Page 3: - Channel 1 to 6 shows the Coil driver Voltage readback of stage 2. - Channel 7 to 12 shows the Coil driver current readback of stage 2. - Channel 13 shows that the ISI master switch was off - Channel 14 shows that the ISI watchdog was tripped. We wonder how the DAQ could drive the coil driver while the ISI real time controllers output were all zeros. Page 4 shows the similar thing for Stage 1.
Here are some Dataviewer snapshoots of the event. Huge excitations start at 11:50am on March 1, 2012. (19:50 UTC)
Failure_3.png presents the coil drivers readbacks. Signals start going out of the coil drivers at 11:50am for no obvious reasons.
Models.png shows the last filter banks before the DAQ. The output channels are (
Failure_6.png shows the output channels of the ISI model and the DAC channels.
It seems that there are discrepancies between the ISI signals and the DAC signals.
There were ETMY M0/R0 excitations beginning around 18:21:00 PST March 4th (2:21:00 UTC - March 5th) March 4th until the early hours of Monday March 5th (local time). For quiet data with no actuation on the ETMY QUAD, any time before March 4 at 18:21:00 PST (02:21:00 UTC - March 5) up to the morning of Saturday March 3 (local PST time).
ISI Coil Driver electronics for both BSC6 and BSC8 are powered down.
Summary of changes made: Last revision prior to changes is revision 1911. To reset Hanford userapps to the old ISI systems, please revert the HPI and ISI common and H2 folders to revision 1911 and re-make h2isiitmy and etmy, h2hpiitmy and etmy, . Revision 1912-1913 are changes to the master model including the new control. Revision 1913 and 1914 are medm screen revision, 1915 is controls on the master model. Revision 1916 includes the changes to h2isiitmy. HEPI Changes: On h2hpiitmy.mdl, I added in three ADCs: ADC0, ADC1, ADC2. These were all terminated. Prior to that, only ADC3 (card num = 3) was in the model. I also changed the bus outputs from the ADC selector, since they were previously connected with arrows copy-pasted from ADC0. No changes were made to the hepi template. ISI Changes: For h2isiitmy, DAC_0 (card_num=0) and DAC_1 (card_num=1) were placed in ISI model - DAC_0 connected to ground entirely since it is shared with HPI. Previously, Lots of controls changes were made: feed forward and sensor correction from 0->1 and 1->2. These new controls all had the outputs from their output filters grounded so that no additional signal would be added to the model until Compiles - The models were compiled at various points to test out an ADC problem - the channels at the input to the ADC did not have the same value as the channels at the input to the HEPI model. At one point, the HEPI model was compiled at 4k to test. In terms of what might have caused the motion craziness, the changes I made were either internal - prior to the overall output watchdog - or external - adding or changing ADC and DAC connections. The watchdog should have still prevented any model changes from driving the platform differently. The changes that might have caused problems with I/O inter-connect problems would have been 1) adding the HEPI DAC to the ISI model, 2) adding the three ADCs (0, 1, and 2) to the HEPI model, 3) changing the connections from the HEPI ADC3 to the hepitemplate block, or 4) recompiling the hepi model at 4k rather than 2k rate to test out the IOP problem we were having. I'll edit/comment on this as more comes to mind, feel free to e-mail me additional questions or comments.
Still investigating, but wanted to break what appears to be the radio silence... These are spectra of the HEPI sensors. Not sure if they're calibrated correctly, but *could* be in nm and nm/s. This unphysically high Q in HEPI points to some digital problem Best (ok, it's the current MIT guess, not necessarily the best) guess is some user model / IOP model exchange problem...
Attached are plots of dust counts > .5 microns. The Comtrol in the LVEA was powered off for a period of time. I have included a plot of H0:PEM-LVEA_DST15_MODE to show approximately when.
Today SUS testers noted ITMY M0 top OSEM sensors railed (see Kissel/Barton alog entries below, 07:47 today). Concerned for the state of the ITMY monolithic fibers, we pulled viewport covers to inspect the quad. While the view of the TM was largely occulted by the AOS ACB, we could see evidence of fiber breakage on the floor in the form of glass shards. The ISI/SUS/HEPI underwent a major and sustained actuation. Investigations are underway to determine the cause. We will open BSC8 early next week to begin repair work. -Landry/Bland/Sadecki
We tried to extract a few notable events: Thursday March 01, 2012 07:40 am PT: - The SEI watch dogs is armed - The SUS watch dogs is on State 1 = "OSEM DC RMS Triggered" (see page 4) - We start seeing more motion in the HEPI and ISI sensors, apparently driven by the beginning of the activity at the observatory 11:25 am PT: We see more motion. The ISI watchdogs trip. 11:47 am PT: huge motion starts being visible on the OSEMS (page 1), HEPI (page 5) and ISI signals (page 6 & 8). It will last until 17:28 PT. 11:58 am PT: The Quad watch dog reaches State 2 "OSEM AC RMS Triggered". (see page 4) 12:24: the ISI real time controller is shut down to recompile the model. It's apparently unsignificant, but we wanted to report this detail. 12:41 pm PT: OSEMS signals suddenly goes to 0. This is apparently when the fiber brakes. (see page 1 to 3) 17:28 PT: the huge motion visible in the HEPI and ISI sensors suddenly stops (page 7) No other similar peak of activity was visible in the past 7 days (see page 11) Main preliminary comments: - there was no drive on HEPI (actuators are being flushed). - ISI and Quad watchdogs are tripped long before the fibers broke - the ISI real time controller was turned down before the fibers broke - unusually high motions started at 11:47am PT and continued till 17:28 PT. -Fabrice/Hugo/Vincent
Just so people don't have to convert time zones for future data mining: 2012-03-01 ~08:00a PT = 2012-03-01 ~1600 UTC = 1014652815 = Morning LVEA activity is on-going 2012-03-01 11:47a PT = 2012-03-01 1947 UTC = 1014666435 = Huge motion starts 2012-03-01 11:58a PT = 2012-03-01 1958 UTC = 1014667095 = QUAD WD trips, AC Coupled WD 2012-03-01 12:41p PT = 2012-03-01 2041 UTC = 1014669675 = Fiber break 2012-03-01 17:28p PT = 2012-03-02 0528 UTC = 1014701295 = Excitation Stops
I have modified the MATLABPATH environment variable to add several new directories for guardian development, specifically /opt/rtcds/userapps/release/*/common/models.
The BSC8 HEPI actuators were valved into the main lines by turning the 4-way bypass valves after the actuators were set in a bleed configuration at 4pm today. The actuators are now in a bleed configuration. The 4-way bypass valves had been set to bleed air out of the main lines before the actuators were valved in.
Electrical work is ongoing - wiring lights, running cables for sensors, etc. AC units now have power, but are not turned on. Next week the acoustic door should arrive, as well as the HEPA units for the clean rooms.
Present status of ISC in-vac components on TMS are:
By the way - we needed about 3.8V to drive the beam diverter. [Two AA batteries connected in series (~3.2V) were not enough.] Rodney measured 14 ohms across the coil.
First look at picomotor wiring diagram, D1100670 page 3.
Yesterday, I and Dan Hoak confirmed that 3 of 6 picomotors didn't work. We disconnected all mighty mouse connectors on the ISC table, and measured the resistance between all of the pins on DB25 cable from outside (i.e. after the mock feedthrough) using a volt meter, so that we were testing three sections of cables (D1000238, D1000921, D1000223) in series.
We've found that 1, 7, 8 and 13 were short-circuited (corresponding to pin 13, 7, 6 and 1 in-vac), and this exactly corresponded to the non-working picomotors.
Today I and Rodney disconnected DB25 on the ISC table, so that we were testing two sections of cables (D1000921, D1000223) in series. Pin 13, 7, 6 and 1 in-vac were still short-circuited.
Then I wanted to disconnect D1000921 and D1000223 so that I can test D1000223 alone, and even before I disconnected anything, as soon as I touched and wiggled the cables on the cable bracket on ISI table, something happened, and we broke one short circuit. Pin 1 and 13 were still connected, pin 6 and 7 too, but these two groups were not connected any more.
I disconnected the cable and Rodney tested D1000223 alone, and pin 6 and 7 were not connected any more, but pin 1 and 13 were still short-circuited.
The pins of the connectors looked fine (look pictures, male (sorry it's blurry) is on D1000921 and female is on D1000223). I dismantled the connector shell on D1000921 thinking that probably I can expose the wires at the back of the pins, but I couldn't because the shield was already firmly climped to the shell (third picture). Anyway, I took out the unused pins (they're useless and make it harder to reconnect).
When I connected D1000921 and D1000223 back together, pin 6 and 7 were not connected any more. WTH? The only difference is that I didn't use much force when assembling the connector shell, and when connecting two connectors.
One explanation is that the insulation of the wires are removed too much or something in the connector shell. If you look at the third picture again, you'll notice that some pins are sticking out much more than the others. Because the back of the wires are firmly bundled together by the climping of the shield, depending on how firmly you press the pins by the PEEK material when you assemble the shell, the wires in the shell could bend differently, making some wires contact with each other.
Anyway, this means that two more picomotors are working now, and we have only one non-functional picomotor. This in itself is not tragic, but it is tragic that we have cable failure which apparently depends on how I touch cables and how I assemble the shells.
It's not clear if the initial failure of pin 6 and 7 was due to D1000921 or D1000223, but we definitely know that pin 1 and 13 is due to D1000223.
Look at page 4 of the wiring diagram. Same story here. Even though it doesn't affect the functionality of the beam diverter, pin 1 and pin 23 in-vac are short-circuited.
Then we started to disconnect things, and when I touched the connection of D1000921 and D1000223, something happened, and no short-circuit any more.
Just for completeness, I disconnected it anyway, didn't find anything (picture 4 and 5), and reconnected them again, fixed it on the bracket, and pin 1 and 23 are short-circuited again. Too bad.
For the record, the serial number of bad cables are:
The only non-functioning picomotor, M1, is used for the steering mirror splitting the green main and the green QPD path. Replacing S1104079, we'll likely regain this important picomotor.
Regardless of this, we might lose M4 again in the future, but this is not critical for one arm test as it is only used for red QPDs.
Fabrice, Hugo,
A GS13 huddle test was performed in order to pursue the investigation of the gain difference observed on the TFs measured on HAM-ISI Unit#2 (alog #2308)
Three horizontal GS13s were tested on the “bench”
Initially installed on this unit. Gain is half too small. One of its two op-amp could be malfunctioning.
Response lower by a factor of 100 to 10000 below 20Hz. The seismic mass can be felt/heard when tilting the instrument, hence it is not stuck. The instrument will be opened at LHO for inspection. It will have to go back to LLO to pass the leak-checking-test before being installed on a Unit.
Locked test-mass got unlocked after manipulation. Acceptable power-spectrum. Not vacuum-compatible.
Conclusion:
Production GS13s are malfunctioning. The Test GS13 Pod #66 will be used temporarily for this testing phase of HAM-ISI Unit #2 testing. It will be removed before storage of the unit. Later on, a production GS13 will be installed for the side-chamber testing.
Test-GS13 Pod #66 was installed on HAM-ISI Unit #2, as H3. GS13 doors were closed. Phase 1 of HAM-ISI Unit #2 testing is in progress. Transfer functions are being recorded overnight.
Due to high wind speeds, the South Bay roll-up door was closed to mitigate the elevated dust levels in the LVEA. Normally, this door is kept open to purge air from the LVEA, but the increased wind speeds overcame the over-pressurised LVEA. Dust monitor levels have subsided to normal levels after the close just before noon.
Correction - The South bay man door (emergency egress) is used to purge dust.
