This morning, the ASC system was found to be blasting signals out after the model change because the inputs and output switches in the main loops were still ON even though all of the Guardians were set to DOWN for maintenance day. So, I set the SDF to capture that the h1ascsafe.snap should restore the INPUT switches to OFF, then the Guardian will turn them on when appropriate later. I watched the guardian turn them on during transition, just now, so all is good.
The AA chassis D1500176 with PI band Pass filter was installed at EX/EY this morning. Units were installed in rack SUS-C2 slot U21. Units installed: S1500143 - EY S1500144 - EX Filiberto Clara
I checked the LTS containers in the LVEA today, I increased the flow to the SUS cluster and included a recent dew point plot.
Chris Soike assisted me with the annual lubrication of the axial vane supply fans today.
Elli, Alastair, Nutsinee
The CO2X IR temperature sensor doesn't behave like it should. I turned the variable resistor while pointing the sensor out to the room until it tripped, turn it back a quater turn to untrip it, put my hand infront of it and it didn't trip. Then I tried 1/8th of a turn (still required a quarter turn from the tripping point to untrip the alarm), still couldn't trip it. Eventually I turned the resistor so close to the tripping point and still was not able to trip it with my hand. We didn't have this problem with the CO2Y IR sensor we installed last week. We are suspecting bad sensor and now looking for a spare. Hope to try again next Tuesday (given the spare sensor is found). The sensor mount has been installed on the viewport.
New ASC model was installed.
SDF showed more than 6000 channels that are not available in the frame. These were either deleted in this model uptate (entire Initial Alignment System, some Alignment Dither System signals etc.) or obsolete channels from long time ago (e.g. all ASC-ALS signals that were moved to ALS model a long time ago).
Betsy made a new snapshot and we confired everything in SDF for new channels.
Note that the ASC MEDM screen is NOT updated. Functionality of the new model is exactly the same as before (new things introduced at LLO are terminated at the top level at LHO except new dither system).
Just after I wrote the above entry, I was looking at the old model file again and caught something that I didn't yesterday: The output matrix signal order was funny (OM1, OM2, TMSX, TMSY, then OM3).
I went back to the new model, and unfortunately it turned out that the new model from LLO "fixed" the funny order by reshuffling TMSX, TMSY and OM3 signals in the matrix. What used to be the matrix elements for TMSX, TMSY, OM3 are now for OM3, TMSX, TMSY, i.e. the TMSX signal is now sent to OM3, and TMSY signal to TMSX.
Since the guardian is not touching this output matrix, we decided to just manually change the output matrix (not the model, but the epics values for corresponding matrix elements). After this change, new output matrix MEDM screen was pulled from svn and everything was fine.
New ASC master medm was also pulled from svn and it looks OK.
I updated the input matrix MEDM, but everything was white on that one so I put the old one back.
And, SDF has been updated to capture the new matrix elements in the ascsafe.snap
A few SDF bugs for the next upgrade please:
1) Today, in SDF I needed to UNMONITOR 2000 channels on H1 LSCAUX. However the SELECT ALL - MON button at the top of the column of 40-per-page does not let you mass-select channels from the the FULL TABLE view. You can only mass-select from the DIFF table. However, if there are no diffs on the channels, they don't show up. Likely this is because the FULL LIST is a mixed bag of monitored and unmonitored channels. Maybe we need to add a MONITORED table to the drop down options?
2) Sometimes channels get moved into the CHANS NOT MON list with or without a setpoint. (See attached.) Probably not a big deal, but annoyingly inconsistent. Today, I moved many of the channels shown in the attached picture to the not mon list and only some of them have a setpoint.
This was necessary as most had somehow been snaped while fully isolated. While at it, most all medms were looked at and unnecessary offsets, switches and settings were zero'd, turned off, and checked. All STATE_GOOD values where available were corrected as needed so these can be used to confirm settings--green is good. GAIN_GOOD/BAD does not work for non unity gains so rely on SDF. All SDFs were verified for appropriate NOT_MONITORED channels and greened. SEI is ready to go.
WP 5427 Old raw minute trend files have been copied from the the SSD RAID array to the /frames directory to make room for continued writing in the SSD array. The h1nds1 daqdrc file has been modified to allow h1nds1 to read the relocated data, this change will take effect when the DAQ is restarted at the end of maintenance.
Looking at timeseries graphs for photodiaodes from two separate locks (7-28-15 8:00:00 UTC for two hours AND 8-1-15 10:15:00 UTC for two hours) I've tightened up all the saturation thresholds in the LSC ODC.
Channels | New Threshholds |
POP_A_RF9 | 21000 |
POPAIR_A_RF9 | 27000 |
POP_A_RF45 | 5000 |
POPAIR_A_RF45 | 5000 |
REFL_A_RF9 | 4000 |
REFLAIR_A_RF9 | 300 |
REFL_A_RF45 | 4000 |
REFLAIR_A_RF45 | 350 |
REFAIR_B_RF27 | 3500 |
REFLAIR_B_RF135 | 350 |
ASAIR_A_RF45 | 4500 |
Updated SDF to reflect these new changes so they will not get lost in future reboots.
This was seen before, and may be related to broken electronics.
The excess noise is visible also on August 1, the first lock after Vern's percussive maintenance. So either the high frequency noise in the OSEMs came back a day later, or it wasn't related. The channels that have the 1821 Hz and harmonics are only recorded at 256 Hz, so I can't check whether the OSEM problem was there or not on Aug 1.
FYI the same coils were oscillating this morning. Satellite box will be replaced.
While looking at a few big glitches, I noticed that many have a curious post-glitch feature around 300Hz. The post-glitch ring seems to match the periscope frequencies we usually see noise at... could this be a hint as to the source of some of our glitches? Maybe this was fixed last night, maybe not.
Looking at all the glitches previously indentified, it runs out that only 17 out of 151 show an excitation of the periscope peaks, as found by Matt. Here are the times:
1117617558.82
1117690781.51
1117713007.28
1117754567.25
1117873501.47
1117888336.85
1118320556.34
1118336224.73
1121762741.96
1121942076.77
1121966658.37
1121986392.35
1121991554.23
1122466633.18
1122474562.14
1122479776.82
1122484101.65
Cheryl, Evan
The 0.43 Hz pitch instability strikes again. I had to turn up the dHard pitch gain from 10 to 20 in order to prevent a runaway. Back into the guardian it goes!
It takes tens of minutes to ring up, and even after tweaking loops, it never exactly rings down (the fuzz in the attached striptool is almost all from this 0.43 Hz feature).
Attached are error and control signals for the arm pitch loops at several different powers: 10.4 W (blue), 15.2 W (yellow), and 19.8 W (green).
Is it strong enough to show up in the suspension OSEMS?
Are the OPLEVs still on? They suppress the ASC gain below 1Hz, and might be the cause of instability.
A trend of the oplev PIT loop buttons over the last 12 hours show the:
ITMY OL damping OFF
ITMX OL damping ON
SR3 OL damping ON
PR3 OL damping OFF
ETMX OL damping OFF
ETMY OL damping OFF
All above YAW loops OFF.
BS OL PIT and YAW damping ON.
THe sttached screenshot is from the few minutes before Evan turned up the DHARD PIT gain last night. You can see that there is an oscillation in the Y arm only, and it seems to be soft. The same oscillation shows up in MICH but it is much smaller.
As Betsy pointed out, ITMX oplev damping was on last night, while ITMY oplev damping was on. Both ITM oplev loops are currently turned on in the INCREASE power state of the ISC_LOCK gaurdian, which still seems to be necessary for stability. Based on OLG measurements of DHARD, the Oplevs should not impact the stability of DHARD. However, we haven't been able to get good measurements of CHARD since rephasing the REFL WFS and hopefully getting a reasonable amount of gain from these loops. (just because of other activities).
Hopefully we can spend some time on CHARD today, which may shed light on why we need the ITM oplevs and hopefully reduce the large optical gain fluctuations we have.
Shivaraj, Darkhan
Summary
We replaced whitening filters on Delta L_{res} output channel with 2 zeros and 2 poles zpk, and Delta L_{ctrl} with 3 zeros and 3 poles zpk.
Details
Madeline uses FIR filters to dewhiten Delta L_{res} and Delta L_{ctrl} in GDS scripts. To make it easier to generate shorter dewhitening FIR filters for these channels she requested to replace the existing 5 zeros and 5 poles zpk filters in both of these channels with simpler, less poles and zeros zpk's.
Shivaraj designed new FIR filters for both of these channels, and we implemented them today around 1pm.
Power spectrum of H1:CAL-CS_DARM_DELTAL_CTRL_WHITEN_OUT before applying new whitening filter plotted in attached "H1-CAL-CS_DARM_DELTAL_CTRL_WHITEN_OUT_z5x1_p5x100_MAG.pdf". After applying new whitening filter, zpk([1; 1; 1], [500; 500; 500], 1), unfortunately the interferometer wasn't stable to take a spectrum measurement after I changed the filter, so I couldn't evaluate "whiteness" of the output in this channel.
Power spectrum of H1:CAL-CS_DARM_RESIDUAL_WHITEN_OUT before applying new whitening filter plotted in "H1-CAL-CS_DARM_RESIDUAL_WHITEN_OUT_z5x1_p5x100_MAG.pdf". After applying zpk([1; 1], [500; 500], 1), the spectrum of the same channel is given in "H1-CAL-CS_DARM_RESIDUAL_WHITEN_OUT_z2x1_p2x500_MAG.pdf".
We also plan to implement similar whitening filers on new (not yet existing) channels for Delta L_{TST} and Delta L_{PUM/UIM}.
Below are plots showing the effect of differnet whitening filters including the one that was being used unitl now (5 x 1Hz zeros and 5 x 100 hz poles). These plots are the basis for new filters for DeltaL residual and DeltaL control. In those plots blue curve corresponds to actual data and other colors represent different whitening filters applied to the data.
With the new filters installed we looked at the DeltaL residual and DeltaL control during a recent lock. Below we have attached the spectrum of both along with DeltaL external as a reference (which is still being used with old 5 x 1 Hz zeros and 5 x 100 Hz pole filter). In the plot the channels are dewhitened with the corresponding filters.
Had the IFO at Engage_ACS while Kiwamu was running some OMC checks. The IFO lost lock, which appears to correspond to a sudden wind gust of ~25MPH. The lockloss plot posted shows this gust.
We did some further investigation using the lockloss tool (alog #: 20337) on this event. The first 48 channels that went wrong before the lock loss were plotted, and a complete list of all misbehaving DQ channels were listed in text file, ordered by time before lockloss. It seemed that we saw a violent ground motion, and lots of coil drievers were trying hard to react on it before they finallyt failed.
==================================================================================================================================================
There was a mistake while I filtering the data into different frequency bands, so the plots I posted yesterday did not really make lots of sense. The error has been fixed and new plots attached.