D. Barker, J. Batch, J. Kissel, R. Lane
After gleefully and boldly declaring victory yesterday on the upgrade to RCG 2.1.5, the CDS gods had to smite us a little bit, just be sure we still new who was boss.
Executive summary:
After some wild goose chases, we achieved the following results for BSFM01 and the X1 SUS Electronics Test Stand (bscteststand2):
- The BSFM is running on RCG 2.1.5, with associated GDS software
- The watchdog behaves as expected
- We are able to drive OSEM coils in all degrees of freedom using awggui, and measure the response with OSEM sensors in dataviewer and DTT.
- We are able to retrieve frame builder data from the past.
Lessons Learned:
- Exceeding the a reasonable amount of the test points (~6) is still bad thing, and can cause all sorts of mysterious badness.
- Therefore when looking at real-time data, whenever possible we should use frame builder channels (_DQ channels)
- A good, full, healthy power-cycle of the front end and I/O chassis is never a bad idea after upgrading the RCG and associated software.
Detailed breakdown of today's activities:
- Robert "prepared" the BSFM using /ligo/svncommon/SusSVN/sus/trunk/BSFM/Common/MatlabTools/Load_BSFM_MEDM_Values.m, in order to restore matrix values, filter bank states, and offsets. (WIN!)
- While trying to un-trip the watchdog, we discovered a few bugs in the BSFM watchdog front end code and MEDM screens, and fixed them. (WIN!)
- Discovered some small, but strange noise in *some* of the test point channels -- not a show stopper (WIN!)
- Once watchdog was reset-able, we attempted to drive small offsets out to the OSEM Coils, using the OSEM Sensors as our figure of merit, without success. However, it appeared that drives made out to the last possible visible point in digital land (i.e. the IOP DAC output channels)
- Went out onto to the floor to further diagnose the problem by measuring the the voltage coming out of the Anti Imaging Chassis, and our suspicions from the OSEMs were confirmed: no voltage was coming out of the Anti-imaging Chassis. While performing this test we noticed some of the indicator lights on the GDS_TP screens were red (specifically the ones identifying that the user model acknowledges the existence of a DAC card).
- Somehow, our CDS woes infected the SEI Electronics Test Stand as well, and they *also* found that they could not excite anything out of there DAC.
- Assuming this was some mysterious flaw in RCG 2.1.x (SUS is at 2.1.5 and SEI is at 2.1.4), we went running to the CDS team. After trying a power cycle of the entire rack, and looking around to see if the timing crates common to botyh systems had anything to do with it (the conclusion was "maybe...?") SUS was able to finally get *some* signals out. Who knows why, our best guess is the power cycling of the rack (though this doesn't solve the same problem for SEI)
- While attempting to get *other* signals to drive, we discovered craziness in the test points in dataviewer (some with huge digital waveforms, others with just a fixed spike, others displaying rediculously huge numbers), and when the same channels were measured in DTT, the were absent (as though the signal were a bunch of NaNs or something). Again to the CDS crew!
- After a few minutes of noodling around, we found that our test points were *all* engaged (as read out by the GDS_TP screen), and our channel datarate had exceeded the allotted 2 MB/sec. Remembering the good 'ol "diag -l" > "tp clear *" method of clearing all test points, this magically alleviated the test point craziness. Note that this *did not* remove the small, 89Hz harmonics noise in the L and T TEST excitation channels.
- I then moved back to this morning's original goal, make sure that all degrees of freedom can drive, and that drive can be measured by the corresponding sensors. This was a success. The subentry below details how I performed that test, and shows the results.