Power supply for the PEM IO chassis was replaced. Unit powered up with no issues. Front end computer was also powered on.
J. Kissel, J. Warner, H. Radkins This completes the changes needed at LHO for the GND STS Kerfuffle, Integration Issue 942, and completes activity for Work Permit 4876. We can't yet close ECR E1400386, or the integration issue, because Ben needs to update the system wiring diagrams, but those should be very coming soon. After moving all STS distribution chassis outputs into the now-pre-defined AA chassis spigots (see drawing in E1400386 for reference, for now, until the drawings get updated), then completing all model changes to include all three corner-station, sensor-correction, STSs in every HPI and ISI last night (see LHO aLOG 14264), Hugh, Jim, and I took on the task of installing the new code. We - used guardian to ramp down each chamber, taking the manager to OFFLINE, where applicable, and where not, bringing each ISI and HPI to the READY state and turning OFF the master switch. - turned OFF all HPI and ISI sensor correction filters by zeroing the MATCH filter bank gain and turning OFF the output switch. - captured a new safe.snap, using ]$ makeSafeBackup ${subsystem} ${modelName} and committed to the appropriate "burtfiles" directory in the userapps repo - *** Made sure all filter files are softlinks to the userapps repo, and repo versions were up-to-date and committed -- we found that the majority of the HPI and ISI filter files (both .txt and .fir) were NOT softlinks to the userapps repo. To fix, we - Copied any non-softlink files to the appropriate filterfiles directory in the userapps repo - Committed the files - removed the files from the chans directory - created the softlink in the chans directory to the userapps repo, e.g. ]$ cd /opt/rtcds/lho/h1/chans/ ]$ ln -s /opt/rtcds/userapps/release/${subsystem}/h1/filterfiles/${filterfilename} ./${filterfilename} - re-compiled and installed the new model code - restarted the front-end code - untripped all watchdogs - cleared any IPC errors on HPI/ISI/SUS by hitting the "diag reset" button on everyone's GDS_TP screen. - used guardian manager to bring up each chamber, takingthe manager to ISOLATED, where applicable, and where not HPI to ROBUST_ISOLATED and ISI to HIGH_ISOLATED - Restarted the frame builder / data concentrator ]$ telnet h1dc0 8087 daqd> shutdown - Confirmed that every corner station HPI and ISI reads out the correct GND STS on the correct channel using dataviewer and me zydeco dancing in proximity to each sensor successively. Note that *** may need some attention when loading in new filters. It's unclear yet what quack (the matlab method of installing new filters that really only the SEI team uses) will do when loading filters. It currently installs new filters into the /opt/rtcds/${site}/${ifo}/chans/ directory. Will it over write the soft-link? Will it follow the softlink? I think this has been checked that it won't break anything, but it's something we should pay attention to in the future. It's especially interesting to see if the .fir filters can handle it. Laissez les Bon Temps Roulet!
Issue likely applies to other subsystems but SEI has a hierarchical guardian and this may be more a problem for these types of guardians.
Before restarting the front ends, the guardian was used to bring things down by moving the request from Isolated to offline. When the BS (and all other) guardian (Chamber Manager) was turned from OFFLINE to ISOLATED after the frontend restart, we had to RELOAD the guardian. This was expected and maybe normal.
However on HAM4, the chamber manager reload was not sufficient. The Chamber Manager reported "ERROR in state INIT; see log for more info; MODE=>RELOAD to reset"
The log file reported "NameError: name 'Connect' is not defined"
Until I reloaded the ISI and HPI subordinate guardians (can't tell if just one or actually both needed to be reloaded), reloading the Chamber guardian was insufficient.
After reloading the subordiantes, reloading the Manager brought things back as usual.
Jeff enlisted Hugh and I to help with a mass snap, model compile, model restart -fest this morning in the corner station for all SEI platforms. The BSC's went fairly smoothly, but when I did HAM5, the watchdogs behaved strangely. I couldn't reset the DACkill or Rogue Exc WD's, but when I randomly tried turning on the master switch, I was then able to reset all WD's. The first couple tries, this immediately resulted in DACkill trips, but the 4th one stayed up. The chamber is now isolating again, but it sound like this should be investigated.
model restarts logged for Wed 01/Oct/2014
2014_10_01 10:33 h1broadcast0
2014_10_01 10:33 h1dc0
2014_10_01 10:33 h1fw0
2014_10_01 10:33 h1fw1
2014_10_01 10:33 h1nds0
2014_10_01 10:33 h1nds1
2014_10_01 10:33 h1psliss
2014_10_01 16:13 h1ioppemmy
2014_10_01 16:13 h1pemmy
2014_10_01 16:35 h1ioppemmy
2014_10_01 16:35 h1pemmy
2014_10_01 19:33 h1fw1
2014_10_01 23:13 h1fw1
unexpected restarts of h1fw1. New psliss model with associated DAQ restart. Recovery of h1pemmy after IO Chassis problem.
Alexa, Kiwamu, Sheila, Jenne This is sheila, accidentally logged in as Jenne We made some attempts tonight to transition DRMI to 3F. We see that we don't have much signal in the 3F detectors at 1 Watt, so we have been transitioning manually to 10 Watts after lock is acquired. We have transitioned PRCL successfully several times, with a gain of 1.5 in the input matrix, this is handled by guardian. We have tried srcl, and used 60% of the 3F with 40% of the 1F. Our next steps are to get some asc loops running, so that we don't spend as much time manually aligning DRMI, and also to try transitioning mich after prcl. The rotation stage has a lot of hysteresis, we will ultimately want some sort of servo to make sure that the input power is what we requested. Kiwamu has measured the modulation depth: yesterday AS RF 90 was 33 counts, today (with the 19 dB RF amplifier and 6dB attenuator installed) it is 270. This was consistent with the OMC scan which were the height of the 45 MHz sideband with DRMI locked was 0.3, today it was 3. (These both mean that the modulation depth increased by roughly a factor of 3). The gains in the guardian have been adjusted to take this into account. The good news is that we have locked DRMI many times tonight, it normally locks within a few minutes. Also, the attached screen shot shows that DRMI was locked on 1F for about 10 hours last night.
A fortuitus lock as we were about to leave let us transition to 3F (at 1 watt input power) , which was stable for about 15 minutes before it dropped (we were aligning) at 7:57 UTC october 2nd. For srcl we used -2 in refl 135 I, for prcl we used 1.5 in refl 27 I, and for mich we used -0.43 in refl 135 Q. ( we transitioned prcl, then mich, then srcl). the rms at the coil output for srm m2 is 3*10^4 counts.
On ITMY, we are testing the blend configuration in use LLO (see 14210 for details). It should help reducing the Z to RZ couplings. The document attached shows the coherence between the Optical Lever Yaw motion, and the degrees of freedom of Stage 1 ISI (measured by the T240s):
- Each page shows 5 hours of data taken between 1am and 6am PT
- The three plots on the left show time series. Top left is the optical lever, middle left is Stage 1 translations, bottom left is Stage 1 rotations.
- The six plots on the right show coherence as a function of time (5 hours on the x axis) and frequency (0.01 Hz to 1 Hz on the y axis). Red indicates a coherence of 1. Blue is a coherence of 0. Top right plot is coherence between Stage 1 X and optical lever Yaw, second from top is between ISI Y and optical lever Yaw... bottom right is coherence between Stage 1 RZ and optical lever Yaw.
Sunday morning: the ISI was controlled with the LHO configuration that has been in use lately (see 14210 for details). There is big coherence between Z, RZ and the optical lever at the microseism and at sub 0.1 Hz frequency. Optical lever RMS over ground motion RMS is about 16.5 rad/m/s.
Monday morning: same configuration, same results and comments. The Optical lever RMS over ground motion RMS is about 15.5 rad/m/s.
Tuesday morning: the LLO configuration is ON. There is no more coherence between Z, RZ and the optical lever Yaw motion (except for some time around 4am PT, maybe the M4.4 at 3:44am in China?). The Optical lever RMS over ground motion RMS is about 11 rad/m/s.
Wednesday morning: the guardian manager has been inadvertendly un-paused, re-engaging the Stage 1 RZ loop, and putting Stage 2 in dmaping only. The coherence is back and the motion ratio is 23 rad/m/s, but this is not a relevant configration.
We have re-put the ISI in the LLO configuration. If the data from the coming night is good again, we'll work on propagating this config to the other units, and updating the guardian parameters.
The last page show optical levers motion on the left (ASD on top, RMS at the bottom), for each of the test masses. Ground motion at corner and end station is shown on the right (ASD on top, RMS at the bottom).
A few comments:
- Though the ground motion is similar at the three stations and in the X and Y directions, the optical levers show very different RMS results. Calibration might need to be checked.
- The frequency content also look very different: SEI not performing the same from chamber to chamber? SUS damping loops not performing the same?
- It looks like there is a 0.3 Hz comb on ETMY. We checked the ISI CPS, they are ok. Optical lever comb?
J. Kissel This continues saga the GND STS Kerfuffle (see LHO aLOGs 14238, 14086, and Integration Issue 942), hopefully close to resolution. Obtaining a verbal approval of E1400386 on the CDS hardware call, with drawings in progress from Ben, but confirmed to be following the drawing I'd put together for the ECR, and getting signatures on Work Permit 4876, I began and finished making the necessary front-end Simulink model modifications such that the corner station, sensor-correction GND STS are read out in a consistent, well-understood order. All corner station HPIs and ISIs have been successfully compiled* with the new changes, and are ready for install, restart, and restore when the integration team is ready (probably in the morning tomorrow). *The only model that I had trouble with was the h1hpibs.mdl. In order to clean up the models, I added some buses and and tags to the DAC output of all models. The bus collects signals from the library part, and ships them to a single DAC. For some reason, while h1hpiitmy.mdl and h1hpiitmx.mdl compile with this bus/tag system, the same system copied-and-pasted from the other two does not work on the h1hpibs.mdl. I was only able to compile by restoring the direct line connection to the DAC. What the french, toast?
I spent most of today dodging the interferometer crew, as they're tasks have changed through the day, and installed ground STS to HEPI sensor correction. It runs, but I have had mixed results getting improvements out of it. I wanted to try this as it offloads some lower frequency Z isolation from the ISI, where it couples to RZ motion. I've had the best results on ITMY, where Fabrice has installed RyanD's blend filters. The attached plots show what I've been able to get. ITMY_before_HEPI_senscor, shows the performance before turning sensor correction on, ITMY_after shows after, ITMX_before_blends shows the performance from 4 days ago, before any changes were made (the ISI was running our standard blends, with STS to St1 sensor correction, and position only loops on HEPI). The first pages are a summary of the seismometer signals, page 2 is the oplev, page 3 is suspension point, and the rest are each of the plots from page 1 blown up.
As far as the ISI goes, pages 6 and 7 of the attached files show the important stuff. Basically, we get good reduction in motion above .1 hz and some amplification below, with ISI or HEPI sensor correction. I think we are doing better at ITMY now, although the oplev (page 2) seems to disagree with me, but the commissioners may have been doing something with the optic when I took that spectra. I'll be waiting with bated breath for the commissioners input, though.
Peter K, Gabriele
We were not able to close the ISS second loop. Every time the board was enabled, the output railed.
We checked that the commands to switch the board on and to select which photodiodes to use are correctly sent to the board. Also the variable gain and the reference voltage are sent to the board. Instead, it seems that the commands sent throught the other DB9 cable (boost on and integrator on) are not received. We don't know if it is a model problem or a DAC problem. However, those two commands are not relevant for the moment.
We discovered that when the board is powered up, the inputs are raised to about -11 V, regardless of the input. We traced the problem to the LT1124 used to buffer the inputs and send them to monitoring channels. Removing the LT1124 solve the problem. We checked that thae same happens with the other ISS servo board we have in the electronics lab. We don't understand this behavior. The AEI board is not bahaving in the same way.
Patrick, Dave
We reconfigured the conlog this afternoon and restarted. The difference between today and yesterday's failed restart is: h1hwsmsr, all channels marked ReadOnly in autoBurt.req (and therefore not in conlog) and new h1psliss model. Patrick is investigating off-line why the h1hwsmsr channels apparently caused conlog grief.
No problems with today's reconfigure, about 2700 channels added (SUS-QUAD_TST) and 58 removed.
HAM6 will continue to be pumped by turbo until associated noise can't be tolerated by commissioners or until pressure reaches 1 x 10-7 torr (whichever comes first) at which point will switch to ion (quite) pump
Jim, Dave
h1pemmy failed at 13:02 PDT, we went to MY and found the IOChassis had not front panel LEDs on, CPU could not see the ADC card. We verified +24V DC power supply was on, it showed 0.2A draw.
Powered down CPU, switched IO Chassis front panel rocker switch OFF, switched rocker ON and IO Chassis turned on (DC current 2A). Powered up CPU, models started, DC draw now 3A.
Had to restart h1ioppemmy one more time to get GPS time correct.
Cause of failure unknown.
Dave, Sudarshan, Richard
LIGOCAM reported that three microphone at EY were disconnected.
I found last week that these microphone would work as normal when switched from their original channel position to some other channel on the AA chassis so, we restarted the H1IOPISCEY yesterday to see if the problem was in the DAC. This was an unsuccessful attempt as mentioned in alog 14225.
Today, I injected 3Vpp signal on all the channels using a function generator and found out the following:
Channel 1-16 are working as normal (I happened to switch those microphones to one of these channels during my first attempt).
Channel 17-30 show nothing on the output end of medm screen monitor.
Summary: The problem is with the AA chassis board. Richard has been notified and this AA Chassis will be switched sometime next week.
Internal cables of the AA chassis were unplugged. They have been plugged in and the chassis has been reinstalled in working condition.
For those interested in looking closer at QUAD model parameters, attached are plots comparing all of the QUAD Main Chains when suspended with wires and also when suspended with fibers. Note, if QUAD data is missing for one of these configurations it's because there was no clean data available. Between the 2 plotted configurations, all 12 (H1, L1, and 3IFO) QUADs are represented. Note, I tried to chose data sets that had the same or similar environmental conditions, but it was difficult due to the fact that some QUADs were reworked on test stands and some were reworked in chamber. In all cases they were mounted on Solid Stack Test Stands or Locked ISIs and in-air.
Data is committed to the svn and can be found at:
/ligo/svncommon/SusSVN/sus/trunk/QUAD/Common/Data/
There does not seem to be a pattern in the data of the 2nd pitch mode peak which are clustered by a specific type of suspension (ETM vs ITM, or wire segment hang vs wire loop hang).
And now with some cursors and in a second format for Brett.
As suggested, I looked at the stiffnesses of the Top Mass blades to see if there is a correlation with the second pitch mode frequency shifts. I don't see it. In order of the peaks on the P to P plot, starting with the lowest frequency to the highest the blade sets used in each QUAD are:
H1ETMx - SET 9 (~1.28Hz)
L1ETMy - SET 13
L1 ETMx - ?
L1ITMx - SET 14/15
L1ITMy - SET 12
Q8 ETM - SET 8
Q9 ETM - SET 2
Q6 ITM - SET 10 (~1.531 Hz)
The blade sets go in order of stiffness from highest to lowest, so SET 2 is stiffer than SET 15. SET 14/15 is a mixed SET with blades still of adjascent stiffness.
I took the two wireloop quads that have the highest and lowest 2nd pitch mode frequencies and made a fit to them. These measurements and their respective made-to-fit models are shown in the attached plot. QUAD06 (H1 QUADTST) is the highest, X1 ETMX is the lowest.
I previously did a fit for QUAD06, see log 14235. The fit for X1 ETMX was made simply by taking the QUAD06 fit and subtracting 3 mm from dn, which works quite well.
Since the outliers are 3 mm apart on dn, the other quads seem to have an even spread between those, and no correlation with spring stffness is evident, then a possible explanation is that our tolerance on positioning the top mass blade tip height is +-1.5 mm.
Attached is a prediction of what +-1.5 mm on dn would look like for the fiber quads.
The black is a model of H1ETMY (which has been the default fiber model for some time) where dn=1.78 mm; blue is the same model but with dn=0 mm; red is again the same model but with dn = 3 mm. Some data is included as well. The H1ETMY measurement is in orange, which matches well because of the previous fitting of H1ETMY. In purple is H1ETMX. I think H1ETMX corresponds to the wireloop quad X1ETMX, which was the low outlier on dn for the wireloop configuration. In that configuration a dn of 0 mm worked quite well to the fit model to the data. Here the same 0 mm dn makes almost as good of a fit. There is not data matching the dn=3mm. +3 mm was found to work well for the high dn outlier wireloop QUAD06, which is not yet a fiber quad.
So it seems that for the existing fiber quads, +-1 mm on dn explains the spread well. However, the most recent 3rd IFO quads, still with wireloops, are the stiffest yet in pitch, so they would be expected to bring this to +-1.5 mm and line up with the dn=3mm red curve.
Posting some notes from recent email converstions looking into the large apparent shifts in dn (top mass blade tip height) and d2 in the all metal build (PUM wire loop prism).
Attachement PUMCOMDetails.pdf is from Eddie Sanchez and is a drawing showing that the position of the PUM wire loop break off in the all metal build is basically the same as where it should be in the final fiber build. However, the model fitting suggests the actual break off is about 1.8 mm lower. So Betsy took some photos of this prism on a suspended metal quad. See image files 1445.jpg to 1447.jpg. Since the prism is round, it could be the wire does not have a clean break off. The pictures seem to indicate the wire has a significant length of a line contact. The 1.8 mm shift could be within this line contact.
The last image, 1449.jpg, shows a picture of the top mass blade spring tip in a suspended top mass. The spring looks pretty well centered, not consistent at all with +3 mm of apparent shift in dn for this quad. Quoting some numbers from Betsy:
"The top surface of the blade, as close to the tip as possible, is supposed to be at 9.6mm down from the top of the bridge notch. The notch is 14.6mm wide, the blade is 5mm wide, therefore the bottom of the blade should line up with the bottom notch. No gauge blocks needed. From the picture, this looks very close to lining up."
I finished preparing the online calibration filters for the DRMI which I started in the last week (see alog 14030). The output spectra are now monitored on projector1 (see alog 14042) in the control room as shown in the above pitcture.
I checked the HSTS and BS suspension filters in the calibration paths in h1oaf. While the HSTS suspensions looked good to me, the BS suspension did not look similar to the suspension model. In particular, the location of resonances did not look correct in the first place. So I changed the BS model by running the BS design Matlab script in the SVN and, copying and pasting the result in foton. The DC gain was then re-adjusted by hand such that it is at 8.85 x 10-5 um/cnts (= 46.40 um / (4 x 217 counts)). Note that this 46.4 um came from DCC-T1100602-v2. And the factor of 4 in the denominator compensates to the Euler matrix in the suspension realtime controller.
The dtt xml file now lives in userapps/release/isc/h1/scripts and is named DRMI_spectra_whiten.xml.
Since all the signals are whitened in the oaf model, the spectra must be un-whitened when they are displayed. I put the anti-whitening calibrations in the dtt file and this calibration must be always applied to all three spectra in order to get the correct displacements. Note that the whitening filters are the ones copied from Livingston at some point in the past and they have the following poles and zeros: zpk([1;1], [100;100], 1, "n").
Calibration factors for the DRMI optica gains:
