ISI code change to ramp DAC drive down when watchdog trips and not instantaneously go to zero volts. Dave composed this and I added in red text. Hugh and Dave. 10 April 2018 WP7462 to implement ECR E1800026 addressing II 9889 with details explained by Stocks & Lantz in T1800031. Summary: we attempted to test the new ISO_RAMP code, but were unable to reach that point due to filter modules outputting very large signals. We abandoned the test and restored the code and hardware back to its original configuration. ---------------------------------------------------------------------------- Hugh elected to test the new code on h1isietmy. Using the latest isi/common/models/isi2stagemaster.mdl (r17117) which uses the new isi/common/models/cushion_DOF.mdl (r17116) and the associated C code isi/common/src/RAMP_ISO_OFF.c (r17084) we compiled and installed h1isietmy. Following a C code review by Jonathan, it was expected that the model would have problems if the ramp time is zero of a divide-by-zero nature. Our first test was to execute the if-conditional block which would run the divide-by-zero lines. While Hugh was manually causing watch dog trips, he noticed that the ISI DAC channels were negatively railing to -10.0 Volts. This was tracked to ST1 and ST2 OUTF filters having a railed output of 1.0e+20 with a zero input. These filter modules have two filters installed: Comp and Sym. We found that the Sym filter (a simple gain stage) was causing the output to shoot to rail on WD trip. It was not obvious to us why this was happening. The rail could be removed by clearing the FM history, and could be prevented by turning off the Sym filter. Are you sure? Hugh's not sure if we ever prevented it... After further setup work Hugh discovered that ST[1,2] RZ T240SUBTRACT_Z filter modules were also railing at 1.0e+20. Due to the violent motion being caused by the ISI DAC outputs instantaneously driving to -10.0V each time the watchdog was tripped, we needed to stop the DAC output from driving the DAC cards. Our first attempt was to SWWD trip the output of the h1iopseiey model. However, the ISI code and/or guardian monitors this and this prevented our test from proceeding. Could not get the guardian to run with IOP(software WD) tripped. We eventually had to go to EY and power off the anti-imaging (AI) chassis for the ISI DAC. We could then green up all the software watchdogs and still be sure that no sudden motion was being driven to the seismic stack. Hugh spent a further hour trying to get the isolation ramp code to be executed, but was being thwarted by the misbehaving filter modules. Same drive to 1e+20 counts & -10Vs. We decided to abandon the tests as maintenance time was running out, and to revert the code back to what it was. I reverted the svn version of isi/common/models/isi2stagemaster.mdl from r17117(05apr2018) to r16398(31oct2017), and isi/common/models/isihammaster.mdl from r17118(05apr2018) to r16422(06nov2017). Thank you Dave! I rebuilt h1isietmy and we were happy to see the DAQ INI file return to its original content. We verified the checksum of the INI file was as it was originally. When the model was restarted, the DAQ-ini-mismatch error was cleared, so no DAQ restart was needed. Hugh then damped ISI-ETMY with no re-occurance of the 1.0e+20 FM outputs. Finally, using the ISI Guardian (no SEI Manager) the stages isolated stabily.
As far as the code problem--Don't have it to look at now since Dave reverted but I suspect some X Y Z etc DOF wires got crossed. Don't know if that addresses the railing FMs but it would not help.
I'm trying to reproduce the issue on our test-stand but I can not.
First test shows the behavior is what I expect - I set the filter module for stage 1, X, to just be a constant (100), gain =1, and turn the filter modules off so that the output of the filter is 100.
Then I trip the watchdog and compare the signal at the filter output to the signal at the output of the ramp stage. I do this 2 times, first with a ramp time of 0, and once with a ramp time of 0.1 sec. In the two attached plots one can see that the red filter module output drops immediately to 0, while the ramp output (blue triangles) either drops immediately to 0 or ramps to 0 in 0.1 seconds. I do not see a big glitch. I wonder what the heck is going on - will follow up with Hugh ASAP.
-Brian
Hey Brian,
We never even got to try the smooth ramp as we could not get the system isolated. We did not think to try a fake isolation state as I think you are indicating.