Attached are plots of dust counts > .5 microns. The Comtrol in the LVEA was powered off for a period of time. I have included a plot of H0:PEM-LVEA_DST15_MODE to show approximately when.
Today SUS testers noted ITMY M0 top OSEM sensors railed (see Kissel/Barton alog entries below, 07:47 today). Concerned for the state of the ITMY monolithic fibers, we pulled viewport covers to inspect the quad. While the view of the TM was largely occulted by the AOS ACB, we could see evidence of fiber breakage on the floor in the form of glass shards. The ISI/SUS/HEPI underwent a major and sustained actuation. Investigations are underway to determine the cause. We will open BSC8 early next week to begin repair work. -Landry/Bland/Sadecki
We tried to extract a few notable events: Thursday March 01, 2012 07:40 am PT: - The SEI watch dogs is armed - The SUS watch dogs is on State 1 = "OSEM DC RMS Triggered" (see page 4) - We start seeing more motion in the HEPI and ISI sensors, apparently driven by the beginning of the activity at the observatory 11:25 am PT: We see more motion. The ISI watchdogs trip. 11:47 am PT: huge motion starts being visible on the OSEMS (page 1), HEPI (page 5) and ISI signals (page 6 & 8). It will last until 17:28 PT. 11:58 am PT: The Quad watch dog reaches State 2 "OSEM AC RMS Triggered". (see page 4) 12:24: the ISI real time controller is shut down to recompile the model. It's apparently unsignificant, but we wanted to report this detail. 12:41 pm PT: OSEMS signals suddenly goes to 0. This is apparently when the fiber brakes. (see page 1 to 3) 17:28 PT: the huge motion visible in the HEPI and ISI sensors suddenly stops (page 7) No other similar peak of activity was visible in the past 7 days (see page 11) Main preliminary comments: - there was no drive on HEPI (actuators are being flushed). - ISI and Quad watchdogs are tripped long before the fibers broke - the ISI real time controller was turned down before the fibers broke - unusually high motions started at 11:47am PT and continued till 17:28 PT. -Fabrice/Hugo/Vincent
Just so people don't have to convert time zones for future data mining: 2012-03-01 ~08:00a PT = 2012-03-01 ~1600 UTC = 1014652815 = Morning LVEA activity is on-going 2012-03-01 11:47a PT = 2012-03-01 1947 UTC = 1014666435 = Huge motion starts 2012-03-01 11:58a PT = 2012-03-01 1958 UTC = 1014667095 = QUAD WD trips, AC Coupled WD 2012-03-01 12:41p PT = 2012-03-01 2041 UTC = 1014669675 = Fiber break 2012-03-01 17:28p PT = 2012-03-02 0528 UTC = 1014701295 = Excitation Stops
I have modified the MATLABPATH environment variable to add several new directories for guardian development, specifically /opt/rtcds/userapps/release/*/common/models.
The BSC8 HEPI actuators were valved into the main lines by turning the 4-way bypass valves after the actuators were set in a bleed configuration at 4pm today. The actuators are now in a bleed configuration. The 4-way bypass valves had been set to bleed air out of the main lines before the actuators were valved in.
Electrical work is ongoing - wiring lights, running cables for sensors, etc. AC units now have power, but are not turned on. Next week the acoustic door should arrive, as well as the HEPA units for the clean rooms.
Present status of ISC in-vac components on TMS are:
By the way - we needed about 3.8V to drive the beam diverter. [Two AA batteries connected in series (~3.2V) were not enough.] Rodney measured 14 ohms across the coil.
First look at picomotor wiring diagram, D1100670 page 3.
Yesterday, I and Dan Hoak confirmed that 3 of 6 picomotors didn't work. We disconnected all mighty mouse connectors on the ISC table, and measured the resistance between all of the pins on DB25 cable from outside (i.e. after the mock feedthrough) using a volt meter, so that we were testing three sections of cables (D1000238, D1000921, D1000223) in series.
We've found that 1, 7, 8 and 13 were short-circuited (corresponding to pin 13, 7, 6 and 1 in-vac), and this exactly corresponded to the non-working picomotors.
Today I and Rodney disconnected DB25 on the ISC table, so that we were testing two sections of cables (D1000921, D1000223) in series. Pin 13, 7, 6 and 1 in-vac were still short-circuited.
Then I wanted to disconnect D1000921 and D1000223 so that I can test D1000223 alone, and even before I disconnected anything, as soon as I touched and wiggled the cables on the cable bracket on ISI table, something happened, and we broke one short circuit. Pin 1 and 13 were still connected, pin 6 and 7 too, but these two groups were not connected any more.
I disconnected the cable and Rodney tested D1000223 alone, and pin 6 and 7 were not connected any more, but pin 1 and 13 were still short-circuited.
The pins of the connectors looked fine (look pictures, male (sorry it's blurry) is on D1000921 and female is on D1000223). I dismantled the connector shell on D1000921 thinking that probably I can expose the wires at the back of the pins, but I couldn't because the shield was already firmly climped to the shell (third picture). Anyway, I took out the unused pins (they're useless and make it harder to reconnect).
When I connected D1000921 and D1000223 back together, pin 6 and 7 were not connected any more. WTH? The only difference is that I didn't use much force when assembling the connector shell, and when connecting two connectors.
One explanation is that the insulation of the wires are removed too much or something in the connector shell. If you look at the third picture again, you'll notice that some pins are sticking out much more than the others. Because the back of the wires are firmly bundled together by the climping of the shield, depending on how firmly you press the pins by the PEEK material when you assemble the shell, the wires in the shell could bend differently, making some wires contact with each other.
Anyway, this means that two more picomotors are working now, and we have only one non-functional picomotor. This in itself is not tragic, but it is tragic that we have cable failure which apparently depends on how I touch cables and how I assemble the shells.
It's not clear if the initial failure of pin 6 and 7 was due to D1000921 or D1000223, but we definitely know that pin 1 and 13 is due to D1000223.
Look at page 4 of the wiring diagram. Same story here. Even though it doesn't affect the functionality of the beam diverter, pin 1 and pin 23 in-vac are short-circuited.
Then we started to disconnect things, and when I touched the connection of D1000921 and D1000223, something happened, and no short-circuit any more.
Just for completeness, I disconnected it anyway, didn't find anything (picture 4 and 5), and reconnected them again, fixed it on the bracket, and pin 1 and 23 are short-circuited again. Too bad.
For the record, the serial number of bad cables are:
The only non-functioning picomotor, M1, is used for the steering mirror splitting the green main and the green QPD path. Replacing S1104079, we'll likely regain this important picomotor.
Regardless of this, we might lose M4 again in the future, but this is not critical for one arm test as it is only used for red QPDs.
Seems like somebody rebooted things and didn't burtrestore h2sustmsyepics.
Left YBM turbo header purge valve cracked open a little -> need to remember to close this before pumping next week
- HAM6 set, dust counts being watched (even as dust monitors are rebooted by Patrick - Apollo using dirty room for degrouting/drilling of HAM5 - Erecting H1-Electronics Mezzanine Storage - Instrument Air being used by GV6 and GV8 - Venting Y-Beam Manifold swapping flange out - EY: Testing to find rubbing - TMS: Looking for replacement cable, Time Domain Reflectometer - Ken working on instrument wiring at H1 PSL - H1 PSL fan units may come today (Friday) - SEI - set elevations of HAM1 support tubes
At around 12:40 we appeared to have lost the ETMy computer. David went out to EY and then performed a remote reboot to get it back online.
Chris and seismic crew did a reboot to the SEI model.
Betsy B., Jeff G. The UIM (L1) LL OSEM flag was suspected of rubbing, so Betsy and I set out to make real-time adjustments/measurements. The first attachment is the final measurement after about 4 or 5 individual measurements taken between OSEM flag adjustments. The UIM LL OSEM flag was taken out completely from the LED beam path, remeasured, then re-seated in the beam path. The extra resonances between 1-3Hz in the L to L TF (red trace) have been mostly remedied, although there is still notable Long. to Pitch coupling. The second attachment is a tuned matlab TF measurement of the Long. to Long. taken a day earlier with added resonances between 1-3Hz on the M0 mass, which was the motivation behind yesterday's investigations. The Long. to Long. measurements are much better after re-seating the flag. More thorough matlab measurements will be run this afternoon to diagnose the remaining DoFs.
Before commissioning starts on ISI-BSC8 next week, the tests and control work realized so far are reported in:
- E1100845-v3: aLIGO BSC-ISI, LHO Unit #1: Phase II Testing Report (Integration)
- E1100294-v2: aLIGO BSC-ISI, LHO Unit #1: Phase I Testing Report (Assembly Validation)
A previous post detailed the new controls that were added to the BSC-ISI master model. These included feed forward (all DOF) and sensor correction (XYZ) from the HEPI L4Cs to BSC-ISI Stage 1, feed forward (all DOF) from BSC-ISI Stage 1 to Stage 2 using a blend of the Stage 1 T240s and L4Cs, and sensor correction (XYZ) from Stage 1 to Stage 2 using the Stage 1 T240s. Other things added to the master model include the GUARD block for guardian development and band-limited RMS filters (BLRMS) that use the Stage 1 T240s and Stage 2 GS13s to calculate the motion of the platform and normalize it on a log scale to the Advanced LIGO requirement. Vincent has wisely added a measurement indicator variable to the top level of the model, so I have added a blinking light to the overview screen to indicate when a measurement is in progress. I've attached two screenshots of the new overview screens to this post. Some concern has been expressed that the screens are too small for the large monitors in the control room, and I am happy to develop a larger version of the screen if more people echo this sentiment. We have also changed the macro inheritance variables that the seismic MEDM screens use. The variables should now be as follows: SITE=LHO, site=lho, IFO=H2, ifo=h2, CHAMBER=ITMY, chamber=itmy, IOP=H2IOPSEIB8, iop=h2iopseib8, USERAPPS_DIR=/opt/rtcds/userapps/release, DCU_ID=75 From these variables, all necessary paths should be available to the user. We are continuing to determine why sharing ADCs between models is weird.
While attempting to add the ADC with the HEPI L4Cs into the BSC-ISI model, we discovered that the signals in both models were not the same - that is to say that the signal received on the same channel and on the same ADC was different between the two models. The signals differ by a somewhat constant factor: the transfer function from one signal on the HEPI model to the same signal on the BSC-ISI model is mostly flat, with a light upturn or downturn at much higher frequencies, and with a certain factor difference that is usually a factor of 2.8 or 1/2.8 (that is to say that the signals are highly coherent, but the amplitudes are different, as if the signal from one was scaled by a certain gain relative to the other). Some signals did match, but the transfer function was not EXACTLY one, as it should be for the exact same signal (there were small deviations from 1 at high frequencies). Since the models are running at different rates, we tried re-compiling the HEPI model at 4k to match the seismic rate. This changed which channels matched between the two models, but there were still some un-matched channels between the two. We are still investigating the source of the problem. In the meantime, the isi2stagemaster model, as well as the individual h2isiitmy and h2hpiitmy models have been checked into the svn should anyone want to look for potential "We've measured the speed of light and it's off by 60 nano-seconds because there's a loose cable" type problems. We've already found one and corrected it, but that certainly doesn't mean there aren't others.
Attached is a DTT screenshot showing the transfer function from the BSC8 IOP (raw ADC signal) to both the HEPI model and the BSC-ISI model for the four horizontal L4Cs on the HEPI (H1-4). The signal appears to be working correctly for the BSC-ISI model (TF is 1 + decimation noise), but is off by a factor of ~1/2.8 for HEPI for H1 and H2 (shown poorly in dashed red and magenta, respectively).
Hoping to get some good quiet time with H2 SUS ITMY after the chamber has finally been pumped down to a few x 10^-7 Torr, I took a quick glance at the OSEM alignment after noticing that M0 Damping loops were not on. Attached are screen shots of the OSEM alignment speedometers; they show that the Vertical/Roll sensors (M0/R0 LF and RT) have either lost their signal, or the flags are occulting the the LED/PD. M0's LF and RT are showing basically ADC noise and "off the radar," R0's LF and RT are at least on the radar (and should still be usable). As such, I've left the system in a state where M0 L, T, P, and Y are damped R0 L, T, V, R, P, Y are damped and from my quick glances at BSC 8 ISI and BSC8 HEPI overview screens, I see that HEPI is OFF ST1 and ST2 Damping is ON One can consider this state to be the case starting at 1014738296 Mar 02 2012 15:44:41 UTC (and should true for at least another hour.)
I did a preliminary trend study with Vincent. Approximately 1:50 pm yesterday (12-03-01-19-50-00), there seems to have been a major shaking incident involving SUS, ISI and HPI. Give or take a few data drop-outs due to computers being rebooted, there were major excursions in many ISI and HPI signals for about 6 hours. The M0 top OSEMs (LF and RT) started going from rail to rail at exactly the same time as the ISI and HPI signals but after about 1 hour, the max min, and mean all went to zero, possibly indicating some mechanical failure. More later - need to check if there are any signs of life on the reaction chain and the lower levels of the main chain.
Attached are plots of dust counts > .5 microns. The counts in the LVEA seem high.
Fabrice, Hugo,
A GS13 huddle test was performed in order to pursue the investigation of the gain difference observed on the TFs measured on HAM-ISI Unit#2 (alog #2308)
Three horizontal GS13s were tested on the “bench”
Initially installed on this unit. Gain is half too small. One of its two op-amp could be malfunctioning.
Response lower by a factor of 100 to 10000 below 20Hz. The seismic mass can be felt/heard when tilting the instrument, hence it is not stuck. The instrument will be opened at LHO for inspection. It will have to go back to LLO to pass the leak-checking-test before being installed on a Unit.
Locked test-mass got unlocked after manipulation. Acceptable power-spectrum. Not vacuum-compatible.
Conclusion:
Production GS13s are malfunctioning. The Test GS13 Pod #66 will be used temporarily for this testing phase of HAM-ISI Unit #2 testing. It will be removed before storage of the unit. Later on, a production GS13 will be installed for the side-chamber testing.
Test-GS13 Pod #66 was installed on HAM-ISI Unit #2, as H3. GS13 doors were closed. Phase 1 of HAM-ISI Unit #2 testing is in progress. Transfer functions are being recorded overnight.
