On Nov 7 I added smoothing to the terribly noisy channels generating the differential pressure. I revert these back to no smoothing.
I updated the EDT frame grabber driver on H1HWSMSR. The ring buffers now work. HWS v1.1.8 now works.
/opt/HWS now points to v.1.1.8 againwe just found out that the tpchn files for the new ALS end station models were not added to the DAQ on Tuesday (though the ini files were). We'll correct this next Tue.
I installed v1.1.8 of the HWS code on H1HWSMSR and it doesn't work (it worked fine at LLO). After some digging, it turns out that the EDT software that runs the camera does not run with 'ring buffers' at LHO. Basically, if you issue the command 'take -N 4' on H1HWSMSR, you get 'Bus error', where this works fine at LLO.
After some digging, I discovered that the framegrabber drivers are v4.2.6.8 at LHO and v5.4.1.1 at LLO. I'm going updating the drivers here and see if that helps.
Here are the changes that I made to H1HWSMSR computer
/opt/HWS that points to /opt/Hartmann_Sensor_SVN/release/HWS_v1.1.8 Run_HWS_H1ITMX now points to /opt/HWS/Run_HWS_X/distrib/Run_HWS_X /opt/HWS so it points to /opt/Hartmann_Sensor_SVN/release/HWS_v1.1.6The following is a message from EDT Tech Support regarding the software problems in response to my query about the software not working on the Hanford machine.
I'm pretty sure this has to do with how the driver is supporting the Linux kernel version (or not) in 4.2.6.8. I see in our changelog, the following line for PDV version 5.3.3.3:
Support for Linux kernels 2.6.18 through 3.2.7 (but excepting 3.0.1.4)
Since 4.2.6.8 is currently loaded on a Ubuntu system with 2.6.32, I am thinking the problem is the driver doesn't quite work properly with kernel 2.6.32. We can't allocate buffers correctly. There might be a clue in the included dmesg output for your "Hanford Machine", but the truth is I don't have the expertise to understand what it is telling me.
I think the bottom line is, if you are running kernel 2.6.18 and later, you'll need to update to a more current PDV driver package.
7:00 Karen, Cris - LVEA cleaning exits
8:18 Betsy - To LVEA
8:42 J. Kissel - To EX to turn on coil driver
8:43 Sudarshan - To EX for Pcal work
8:58 Andres - LVEA for 3IFO work
8:58 Doug - To EX to take pictures
8:59 P. King - To H2 Enclosure
9:00 Bubba - To LVEA Moving SEI stuff
9:25 Andres - Back
9:27 J. Kissel - Back with fixed ETMX sensors
9:39 Doug - Back
9:40 Travis - To LVEA to meet up with Betsy
9:47 Travis, Betsy - Back
9:58 Sudarshan - Out of EX
10:06 Bubba - Back
10:39 P. King - Back
11:47 R. Savage, Nutsinee - To MY for check on storage
At ~20:15 on Feb 4th, Hugh switched the GS-13s on ETMY ISI to low gain, and these haven't been switched back since. It's not clear if this impacts the performance of the ISI or not. I've attempted to find long stretches where ALS or oplevs were available to say something about the performance with out of loop sensors, but so far I haven't been able to get a clean before AND after spectra. ISI local sensors don't show any difference (see my plots earlier today showing ETMY vs ETMX suspension point motion, ETMX has always been in high gain). I attach a trend from the 4th to the 6th showing ALS CTRL and one of ETMY's GS-13s. The gain switch is pretty obvious on the GS-13 plot. The best I've been able to find is my second attachment, showing ALS CTRL spectra, dashed lines are high gain, solids are low. Only differences I can see are likely due to higher ground as the solid lines were at 21:20 (3:20PM local). ITMY's GS-13s are in high gain for both measurements, and so could be obscuring this measurment. I'll try to get a measurement with both ISI's in high and low gain.
Jim, Krishna, Ryan, Jeff, Sheila, Alexa, Evan, everyone
We have been having trouble keeping the Y arm locked with green, due to cavity motion of about 10 um in about 10 seconds. The winds at end Y are approaching 40 mph, and have been climbing for an hour and a half. We have copied the 90mHz blends for X and Y from the ETMX ISI into ETMY and blending high has allowed us to lock ALS for now.
The attached screen shot shows the Y arm control signal (from the end station PDH with no slow feedback on) in beige, and the X arm in yellow. The blends were switchd at about -21 minutes.
Here are a few more PEM screenshots of the wind and what it has been doing.
J. Kissel, S. Dwyer For the record, Sheila only remembers changing the blend configuration for ISI ETMY, which is why only the brown trace (ALS-Y_REFL_CTRL) gets visibly better in the StripTool.
The attached plots show:
J. Kissel ETMX UIM Driver's rocker switch tripped last night at 8:44:30 UTC (Feb 06 2015 00:44:30 PST), which cause all OSEM output on the L1 / UIM stage (and the sensor signals as well) to fail. I've turned on the rocker switch, and the stage appears functional. Other points: ------------- There is no smoking gun for this chassis power outage. There are not enough diagnostic tools in the frames to look at all possible suspects (power surge to the chassis [no power monitors], chassis overheating [no temperature sensors], an inconsequential component on say -- a monitor board -- smoking [only one of the four monitor signals for each channel are stored], too large a current request [the NOISEMON is stored, not the FASTIMON]). Output request of DAC (from any ISC) does not appear to coincide with trip, but difficult to tell with dataviewer. There is an Longitudinal request to drive really hard, but it's unclear whether it's a the cause of or a result of the driver's power being shut off. Output request is roughly 42 [mA] (according to FAST_I_MON) on all four coils leading up to trip, switch trips on 3 [A]. Even the sudden, very large output request does not cause any current larger than the 42 [mA] -- as reported by dataviewer / EPICs. Reqrettably, only the severely-high-passed-at-5-[Hz] noise monitor signals are stored in the frames, but I attach the time series of one of the coils anyways to indicate the exact time. It shows a nice smooth ramp down at the time of the chassis power down. Serial number of this box is S0900303. I attach pictures of the box after I'd turned it back on. When I first approached the box, the front "power" LED lights were OFF, and the rocker switch was in the opposite (powered off) position. I'm not sure if there is good way to make this power shut-down control-room visible. I did notice that all the coil driver monitor signals were frozen at -16 [ct], and that the OSEM sensor speed dials were zero for this stage and this stage alone, which is what immediately clued me in that it was a flaw with the entire coil-driver chassis or satellite amplifier. This last happened (according to collective analog CDS crew's memory) on H1 SUS ITMX UIM driver, some time ago, 19 November 2013. See LHO aLOG 8635.
What's going on here?
Adding more to Daniel's comment -- the problem in the screen shot he shows is ONLY a problem with the reported decimation filter output for the ETMX L1 LOCK L bank. He had repeated the same test on the ETMX L1 LOCK P bank, and saw no difference between the OUTPUT and OUT16. It's not an issue progressing forward because no one and nothing uses the decimation channel but for diagnostics, but it's this kinda of stuff that makes users immediately suspicious of the entire front end code as a whole (regardless of how serious the problem actually is) and demand "REBOOT! REBOOT!" without really identifying the specific problem. Again, all signs point to that this is UNRELATED to the coil driver switching off from a current overload. This being said -- I agree the decimation display problem with *this* filter bank *is* weird, and we'll try to debug further.
I have seen this too. Good to know its not just in my mind.
A couple more observations from bizarro land:
As indicated quickly by Daniel (LHO aLOG 16604), I restarted the front-end process for h1susetmy Tuesday morning, and saw no change in this behavior. Since this bug seems to be extremely low impact, we should toss it into the CDS bugzilla for exploration of the flaw offline on a test stand to see if we can reproduce the error and hopefully debug. Also -- just because we like to blame everything new, it might be worth a check to see if this bug appeared after the RCG 2.9 upgrade, or after Daniel's Tidal Servo installation (see LHO aLOG 15601, or from his new Integrator filter module (LHO aLOG 15560), which is installed and feeding to/through the UIM L filter bank.
Ed, TJ, Kiwamu,
We fixed an issue associated with the camera servo in the green ASC loops. It is now functional as it should be.
When we were going through the initial alignment sequence this morning, we noticed that the Y arm green transmitted light decreased after the ASC loops were engaged. We figured out that this happened only when the DOF3 (which is the ITMY camera servo actuating on all three suspensions) was engaged. Adter some inspections, we found that the input of M0_LOCK_P(Y) were off on ETMY. Since the camera servo is the only one who actuates on ETM, the servo was introducing some funny imbalanced feedback into the cavity. Bad. We enabled them and confirmed that this fixed the issue.
Also, Alexa told us that she had a similar issue in the X arm green ASC yesterday and she had to switch off the camera servo. Checking M0_LOCK_P(Y) filters in ETMX, we found them also disabled. So we enabled them. We did not check whether this fixed the situation or not, but I believe that this fixed the issue in the X arm as well.
Alexa, Kiwamu
In response to the commissioning alog from the last night (alog 16501), we spent an hour or two to study the DRMI lock acquisition with ETMs miasliged in this morning. The key item we checked this time was the BS limiter. We took a few lock acquisition samples with and without the BS limiter which was suggested by Ryan.
It seems that the limiter improved the locking time. We will tentatively keep the limiter on.
The alignment was adjusted initially so that we observed consistently high build up in each lock. We tried repeating the same test with the arm cavities aligned, but we ran into some aligment issues which prevented us from completing the test with the arm cavities aligned. Note that this limiter method is something we used to use in the past, but apparently we decided not to use it for some reason at some point.
(The test without the limiter)
| time to lock [sec] | |
| attempt #1 | 10 |
| #2 | 120 |
| #3 | 540 |
| #4 |
270 |
(The test with the limiter)
| time to lock [sec] | |
| attempt #1 | 5 |
| #2 | 120 |
| #3 | 60 |
| #4 | 80 |
| #5 | 90 |
(The limiter)
The limiter resides in BS_M3_ISCINF_LIMIT and the value was set to 5x105 cnts.
This limiter was being engaged during lock aquisition and then turned off when lock was aquired during the MICH_DARK_LOCKED state of the LSC_CONFIGS gaurdian. However, none of this was translated over to the ISC_DRMI guardian. I have added this to the ISC_DRMI guardian now.
Attaches is a plot corresponding to the test that Kiwamu and Alexa did. Once the SWSTAT is around 46000, the limit on the BS correction was turned on at 500,000, and the DRMI started acquiring lock more frequently (examples of locking attempts with this limit OFF and ON are attached as well). Since this setting was not in the guardian managing the DRMI lock (and the MICH guardian used in the initial alignment sequence turns this off once lock is acquired), the BS correction limit was typically off, unless someone was intentionally turning this on (which I am pretty sure happened at some point in the past, as this limit is a known important variable for the locking sequence). How critical this BS limit is for lock acquisition it depends on the seismic input. So, even assuming comparable good alignment states, the easiest explanation for "DRMI lock sometime works, sometime it doesn't" is that even when the seismic noise is not outrageously bad and it is obviously recognized like a problem, it might still be higher than "normal" and make the locking more sensitive to gain settings, and marginal. Here is some (now old) systematic study that Den did sometime ago at LLO changing the locking thresholds, also known to be critical parameters. It would be good to do similar tests with arms off-resonant. I think Alexa and Kiwamu tried by they didn't have enough time today, we should keep this in mind and use any available "free" time to make the acquisition time as short as possible ( < 1 min). Also, it would be good if changes to whatever "nominal" settings were clearly highlighted and somewhat motivated, so we can try to make sense of what's happening.
Sheila, Elli, Evan, Ryan, Lisa Apparently today was a particularly bad day for the whole locking business. To get the DRMI locking with the arms off-resonant, we had to retune the locking gains as previous settings were not working, meaning we could not get a single lock of DRMI + off resonant arms in ~ 20 min. After carefully aligning the DRMI (with no arms), we changed the locking settings as follows: DRMI ---> DRMI with off-resonant arms MICH 3 ---> 3 PRCL 15 ---> 8 (it was 11) SRCL -50 ---> -40 (it was -45) It would be extremely helpful to test these new settings again several times, and do some systematic lock acquisition studies from DRMI to DRMI with off-resonant arms in the morning. It still takes too long to acquire lock. We then went through the locking sequence, and stop at some point to give Evan and Elli a chance to measure the opto-mecahnical TF of the DHARD alignment DOF, so we could design a good filter to increase the bandwidth of DHARD, if necessary. We broke the lock in the process, and Sheila struggled to get ALS DIFF working right after that. While investigating the problem, Ryan found out that we could not get any actuation at all from the ETX L1 stage. Please investigate/fix this in the morning, a manual reboot at the end station might be required. On a positive side: * Evan and Elli got a good measurement of the DHARD plant and designed a compensation filter for that, so we should be ready to test it; * ETMY L1 has now a working L2P filter, so we could eventually try DARM actuation to both end masses, if necessary.
What new ETMY L2P are you referring to?? We found no filters engaged and a gain of 1.0, and the output OFF.
The tidal feedback goes to L1 state. Locking the Y-arm on green and engaging the slow servo caused a really bad L2P with the above configuration. We did a burt restore to yesterday morning (L1 L2P FM8 z0.5p10 gain -1, output ON), and everything is fine now.
H1 ETMX L1 / UIM Actuation / Sensors restored by turning the chassis back on (flipping the rocker switch on the back). The chassis power switch had tripped due to over current. See LHO aLOG 16511 for further information.
The EY L1 L2P state that I found this morning is not what Ryan designed last night ( FM9 (zpk([],[0.5],1,"n")gain(0.001)), Gain 1 and output ON). However, this does not work with the slow servo on.
The EX L1 L2P FM9 was installed zpk([],[0.5],1,"n")gain(0.001). The overall filter gain remained 5.2. Not much invesitgation has been done, but based on the green transmission dips, it seems that the old FM10(-60dB) setting is better.
Elli, Evan
We have taken a TF for the DHARD pitch plant. Now filters are installed in ETMX/Y L2 LOCK P which invert this plant.
With the DHARD loop open (and oplev damping engaged), we drove band-limited white noise (up to ≈4 Hz) into DHARD EXC, which drives the ETM PUMs differentially in pitch. We then measured the transfer function DHARD EXC → DHARD IN1. This gives the plant transfer function for the DHARD loop, which we then fit by eye in Matlab to a zpk model.
The fit is not so great in phase above 0.5 Hz, but maybe as a first try it is good enough to allow us to bump up the DHARD bandwidth anyway.
The filter modules are FM6 and FM7.
Earlier today, with Ryan's encouragement, I turned on more of the I/ETMY isolation loops. Currently these 2 ISI's are running a configuration like LLO's, with all DOFs engaged on St1 and X,Y,Z,RZ running on St2. The St1 RZ loop is running a high (750mhz) blend with no T240. St2 is running the same 250mhz blend on Z and RZ as it was previously running on X and Y. The Stage Guardians for these two chambers have also been modified to turn on these loops, in case we have some traumatic event.
FF is now running on all test mass chambers as well with X&Y running on ETMY, Y on ITMY and X on E/ITMX.
Turning on the extra loops seems to mostly reduce the suspension point motion, although vertical motion looks to be a bit worse at .1-.3 hz. Attached plots are ETMX and ETMY suspension point motions, respectively. ETMX was running the normal LHO configuration, with no RZ loops on, only X&Y on stage 2, LLO blends plus ST0-1 FF running on the Y degree of freedom. ETMY was running a mostly LLO configuration, with all loops on St1, LLO blends plus the Start blend on RZ, St2 with all loops running except RX & RY, with St0-1 FF running on X & Y.
Jim, Is that "start-blend" still using just the L-4C as the inertial sensor? If so, it is going to be noisier than you want down below 1 Hz. Let's look for a similar high-blend which uses the T-240s.
Plots of the two high blend filters we have. CPS Signal is common to both, blue is inertial (L4C only, no T240 in that blend) part of the Start filter, green and brown are inertial parts of the T750 filter.
The smoothing factor was 0.75 and is now 0; see the attached for how that affects the channels.