The remote power control unit in the right hand DTS rack (the one with dtsboot in it) was given the ip address 10.11.0.100 and put on the dts network. It controls the power to both supplies in dtsfe2,3 and also the Dolphin unit. It can be accessed by port 8100 on badger.
I fixed the problem with awgexc_run for the long signal injections that Vincent has been trying to run using Matlab. The issue was that the SIStr library of routines used by awgstream, awgexc_run, and multiawgstream was eventually attempting to send data to the awg too soon. The awg has a limited amount of memory that it uses to store a waveform - the trick is to send data fast enough that the awg doesn't run out of data, but not so fast that the awg's buffer would fill up. When using the SIStr library to send data to the awg, the library loads the awg buffers full, then before attempting to add more, the library calculates a "sleep" time that is needed to wait before trying to send additional data. After close to an hour of adding data, the library miscalculated and sent the data too soon. The error was not handled gracefully, causing awgexc_run to exit. I fixed the problem on the test stand by shortening SIStr's lead time, to keep it from sending data to the awg so far in advance. I will be examining the data I collected in debug mode to see what the trend of the time calculation is (for possible error accumulation). I will also see if SIStr should be changed to handle the case of attempting to send data that is "too far in the future", as this shouldn't necessarily be a fatal error if the waveform data is part of a long stream of data and it's possible to wait a bit longer to resend the data.
We've noticed that quite a few (5 of the 16 used today) BOSEMs had the screws loose which hold the connector to the body. It was noticeable when plugging in the connector to the head and watching the whole connector wiggle.
After returning the sat boxes to the rack, M0 F1 was glitchy. Filiberto reseated the cable at the Sat box which appeared to fix it. Now hours later, we are aligning the BOSEMs and have found that this signal peggs at ~6k alot. We have been able to fix it by reseating the cable at the sat box, but this is not working right...
M0 F1 still a problem this morning. Filiberto thinks that the Sat box looks like an OP amp blew again - he's pulling the box again for a trip to the ER, er I mean EE.
Filiberto returned the sat box after another Op Amp swap, but a few hours later we now see bad BOSEM signals in this module...
Day 5: Still working on diagnosis of crazy signals from the Sat box 1 (M0 F1, F2, F3, L). With Filiberto, performed another round of boots to the AA chassis, and bscteststand computer. No changes so far. Likely another blown op amp on Sat box 1. Since we seemed to have changed both satellite boxes and a set of cables (put the new Jay cables in), we might have possibly had more than one problem to diagnose. We decided to put back in the old cables on a different sat. box and things looked pretty good. Filiberto is going to fix the blown op amp on the Sat box 1.
After rearranging sat boxes, Richard and Filiberto found 2 that seem to work. They hauled away those that have issues. With the working sets, I finished aligning all 6 M0 Top BOSEMs. I just have to finish setting them to 50% OLV and then we can start FFTs.
Betsy, Here is a summary of the troubleshooting of the Sat units. Oct 8, 2010 Uninstalled two Satellite units SN320073 and SN320078 from test stand. SN320078 had two bad channels. CH1 had gain off by ~6dB. Replaced IC103 (OP2177AR) with OP284. Did not have OP2177AR replacement part but has been placed on order (expected arrival date 10-14-2010). CH3 had output railing high, replaced IC303 with OP284. SN320073 had CH4 output railing high. replaced IC403 and IC404 with OP284. Ran test on units per Test Plan T080062 - UK Satellite Amplifier Pre-Preduction Test Plan for transfer functions and voltage readings. Oct 11, 2010 Reinstalled SN320073 and SN320078 in test stand. Took opportunity to test unit SN3200715. Tested unit per Test Plan T080062 - UK Satellite Amplifier Pre-Production Test Plan. All test results within specs. Reinstalled unit back in test stand. Oct 12, 2010 While connecting OSEM's Betsy and company found MO F1 signal to be glitchy. This corresponds to unit SN320078 CH1. While trouble shooting, switched cables with another sat unit. When reconnecting cables to original sat unit, CH1 not responding, output ~2.3V. Uninstalled SAT unit SN320078 and placed a working unit in its placed. Replaced the "new" cable and installed an older cable on the vacuum side. Connected all four OSEM's and saw a response from all four channels. Disconnected SAT unit and left all four OSEMS connected. Replaced IC103 with OP284. This IC was one that had been previously replaced. Ran transfer functions on CH1 per Test Procedure T080062. Unit tested good. Used spare cable and OSEM SN0011 from Betsy to test connecting and disconnecting osems while sat powered on. Trying to see if the act of plugging and unplugging the OSEM could be causing opamps to go bad. Oct 13, 2010 Reconnected unit SN320078, knowing that all four OSEMs were connected, we expected to see all four channels come up. Upon powering unit up, all four channels seemed glitchy until CH3 died. All other channels then settled to a good state. Unit SN320073 also had CH4 fail again. Oct 14, 2010 Replaced IC103 (CH1) and IC303 (CH3) on unit SN320078 with OP2177. Replaced IC402 and IC404 on CH4 with OP2177. All opamps that were continually going bad were the OP284. Since they were being used as substitutes until OP2177 arrived. Oct 15, 2010 Took readings of OSEMs. SN # Our Readings Stuart's Reading Symptoms SN688 55.44 57.6 Working Unit SN695 61.08 62.67 Glitchy - Case causing short SN674 48.99 49.51 Low Count SN697 54 55.37 Working Unit - 29K Count Data corresponds with Stuart's data. Even OSEM SN674 shows a lower voltage reading that is consistant with the low count reading from MEDM. Looked at the bad OSEM with the short. The short is coming from the Photodiode Cathode side. Also, we connected and disconnected various OSEMs and don't seem to be having the issue of blowing up the Opamps. Still not sure how the first set of Opamps went bad.
After removing them from the stand last Thur, 2 Sat boxes were modified to fix a blown IC chip (OP2177). Symptoms were that output of IC chip were railing. Replaced with OP284, while waiting for correct OP2177 which are on order. Likely the Sat boxes will have to revisit the EE shop when these chips are in later this week. We will progress with BOSEM alignment and testing until then. Filiberto reinstalled the 2 sat boxes into the test stand this morning.
We completed the test assembly of the Beam Splitter Upper Structure. To ease installation of the Shear Plates we enlarged all but the top two mounting holes to 29/64”. Beam Splitter Upper Structure Assembly Procedures: (1). Assemble the Upper Structure Face Plates, Bottom Ring and US Cross Member. Snug the bolts enough to allow a semi-ridge structure, but with enough give to allow some wiggle. This will help to align the Face Plate bolts. (2). Stand the assembled Upper Structure upright on a level surface. (3). Hold one of the Shear Plates to a Face Plate and run in the top two bolts. Lightly snug these bolts, but do not tighten. (4). Run in the remaining bolts so the washer has about a 1 mm gap between the Shear Plate and the Face Plate. Note: Rock the structure from side to side slightly and/or wiggle the Shear Plate to allow alignment of the bolts. When you get to the bottom bolts, it may be necessary to lever up the Shear Plate due to sag. (5). Repeat steps 3 and 4 on the other Shear Plate. (6). When both Shear Plates are mounted, tighten the bolts using a crossing pattern, paying close attention to squaring the Shear Plates to the Face Plates. Make at least two rounds of tightening to ensure the structure does not creep, as the bolts are tightened. (7). When all the Shear Plate bolts are tight, tighten the bolts on the Bottom Ring and US Cross Members. (8). Remove the Shear Plates and store until the Beam Splitter has been installed in the chamber and aligned. This procedure should ensure the Shear Plate to Upper Structure fit is correct for installation. If it is not, you may try slightly loosening the bolts on the Bottom Ring allowing the Structure to “float” into a square alignment. Tighten all bolts as noted above.
Syptoms before plugging in any OSEMs - medm actuation verified to be disabled. OSEM signals strange in 3 of 12 signals (-4k, -31k, -19k, when should be zero). Fault lights on 3 of 5 sat boxes. 2 of 2 yellow cables tested out ok. Even unplugged, signals strange. Powered bscteststand down from work station. No go. Powered it down at computer (button) Powered down Expansion chassis. Powered on chassis Powered on bscteststand Ran startup script Richard poked around Powered everything down again Reseated ADC More troubleshooting at the rack by Richard/Filberto... Pulling 2 satellite boxes to take to the EE shop.
After a report form Vincent that the time had drifted enough to make awgstream unusable, I modified the configuration files on seiteststand and workstation for ntp. On seiteststand: 1. stop ntpd 2. modify settings in configuration file 3. sync time with ntp server 4. start ntpd On workstation: 1. stop ntpd 2. Modify settings in configuration file to use seiteststand as time server 3. sync time with seiteststand 4. start ntpd Checked bscteststand, suswork1, found the time had drifted on these as well. Modified the configuration files on bscteststand and syswork1 similar to seiteststand, workstation. Checked times, they are sync'ed again. Will monitor this, seiteststand was stubborn about being able to connect to the ntp server although bscteststand had no problem.
After monitoring the clock performance on the workstation connected to seiteststand all day, I decided to set up a crontab (as root) to run 'ntpdate -b 10.11.0.24' with the output going to /var/log/ntpdate-cron.txt to see if this will keep better time. Will revisit in the morning.
See images for final positioning work. Great job JimW EricA MitchR! Testing will begin on Unit#3 after final instrumentation and cable routing.
Last week we swapped QUADs out on the mechanical test stand. This never being done before, it took a few days to work the kinks out of maneuvering 4 sections of QUADs simultaneously. Currently, the Q1 upper structure is in it's case waiting for an N2 purge, while the lower structure is parked in the modified LSAT Top Plates on the "installation Genie". It's case will not arrive for a few weeks yet. Q2 was fully hung earlier in the week. As of today, all 6 DOFs have been adjusted mechanically so we are installing the tablecloth and BOSEMs. A final tweak to pitch will likely be needed during the OSEM installs over the next day or so. (Note our alignment is to within a few mRad in pitch with the 5mm gap between chains set with a 5mm spacer.) The chains are slightly misaligned laterally (side shift) with the barrel of the ReTM set <1mm off the barrel of the TM. We could adjust this out at the expense of overall height of the Test Mass. We opted to live with the slight side shift for now. Q2 has one Top Stage Blade cartridge which is almost out of yaw range in one direction. The screws are very difficult to adjust under normal circumstances, but one of the outer most blade clamp 3/8-16 is feeling very rough when loosening. The yaw which is somewhat pinned is the motion of M0 in the counter-clockwise direction as viewed from above.
I will be applying some critical security patches to the aLOG system tomorrow around 8:00am. The system will require a reboot. Downtime is estimated to be on the order of 5-10 minutes. Please remember to save any unfinished entries as a draft prior to 8am pacific tomorrow.
The system has been restarted. The maintenance work is finished.
Added new Linux workstation to SUS Quad test stand. It's name is suswork1. When logged in as user controls, the home directory is the same one as on bscteststand. You should be able to run programs on suswork1 the same as you would on bscteststand without having to ssh to bscteststand first. Matlab is installed locally on suswork1. Adobe Reader is installed, and firefox is set up with the aLIGO LHO Logbook as a bookmark. Be sure to use your own name for logbook entries! All /cvs/cds/llo files are visible on suswork1, as are all /apps/Linux files. Details - mounted bscteststand:/cvs, /apps, /opt, and /home/controls to suswork1:/cvs, /apps, /export/opt, and /export/controls. Made symbolic links from /export/opt to /opt for directories in /export/opt. Set up shared library default paths to the same as on bscteststand. Installed or updated libraries to run gds tools. Set up suswork1 to ntp sync time from bscteststand. Let me know when you find stuff that doesn't work.
We have completed assembly of the first Beam Splitter Lower Structure. No fit, machining, or alignment issues were noted. The main Lower Structure drawing (D080005) calls for eight 1/4-20x0.5 SHCS. However, the EQ Stop Plate (D080008) and the LS-US Bracket (D080011), both require 1/4-20x.05 SHCS and therefore 16 of the 0.5” bolts are required.
Over the past two weeks we have repurposed the three X2200 units for aligo auxillary system prototyping. From top to bottom the X2200 are being used for: top: 10.11.0.70, Cyrus network management tests middle: 10.11.0.72, Patrick aconlog prototype bottom: 10.11.0.71, Cyrus second net mgt machine
