Due to an issue currently under investigation by Jamie, the Guardian nodes for BS, ITMX, ETMX, and ETMY were restarted at ~10:30am PST. The SPM diff monitor was reporting various SWSTATs were being toggled quickly (a few times per second), but the SUS MEDM screens reported no such behavior.
Prep for HAM 6 vent ongoing
Jim W. working on filters for ISIs
HAM 6 SUS TFs have been taken in prep for vent (look good), SDF work ongoing
CDS installing dewpoint monitor cables in H2 output vicinity
Purge air being turned on
2nd cleaning on cleanroom complete
OpLev working on replacement lasers, PR3 needs recentering
PLL work today
Jeff B. moving dessicant cabinet into high bay for craning tomorrow
PCal work Tues. at EY, maybe EX
I worked on the output optics ASC characterization tonight. The measurement details will come later.
Here I want to note my activities for the other IFO users who might find my mess:
1. Initially I tried to use ITMY single bounce (PRM/SRM/ITMX/ETMX/ETMY misaligned). In reality, I had many difficulties having AS_C, AS_A_DC, AS_B_DC, OMC QPDA, and OMC QPDB in their linear regions at the same time. I jiggled SR2, OM1, OM2, OM3, and OMCS alignment biases. but no success. In the end, I gave up to use the single bounce.
2. I returned to the original values of the bias sliders. Decided to use DRMI which seemed reasonably aligned. I had to slightly tune the alignment of the BS, PRM, and SRM to acquire the lock.
3. Once lock is acquired, BS, PRM, and SRM were slightly aligned to maximize ASAIR_B_RF90 on the wall striptool.
4. In deed, all the QPDs start to show somewhat linear signals. The SRC2 ASC servo was used to align the beam on AS_C. This servo output was eventually offloaded to SR2/SRM.
5. After spending some time on the OM1-3, OMCS alignment, the beams were mostly center of the five QPDs on HAM6. Only AS_B_DC Pitch had residual of -0.3~-0.4. I decided not to go further. During this process, ASC output matrix for DC3/DC4 were touched. (and returned to the original values).
6. OMC was locked on one of the sidebands. The transmission was ~20mW. This was comparable output power to the DC readout.
6. TF and PSD measurements between SRC2, OM1, OM2, OM3, OMCS to the five QPDs and the dither errors. During this measurement, SRC2P/Y FMs were touched.
7. OMC Dither ASC signal seemed to convey no ASC information. I tried several frequencies and retuend to the original freqs (1.68k-1.75k) with dither amplitude increased from 50-200 to 1000.
Dither demoduration phases were moved from (90, 90, 90, 90) to (75, 90, 90, 100).
8. Then moved on the OMC dither LSC characterization.
9. After some cleaning up, I returned to the single bounce. After all, the ITMY single bounce beam shows up all of the QPDs around the center. This is good. So I decided to leave this alignment. We want to recover this state during the HAM6 vent next week.
10. IFO is left with DRMI and LSC not engaged
Alignment sliders before and after:
PRM (P: -1369.8 Y: 506.37) -> (P: -1361.7 Y: 492.4 )
BS (P: 149.03 Y: -297.35) -> (P: 148.98 Y: -298.04)
SR2 (P: 1935.2 Y: 776.7 ) -> (P: 1936.0 Y: 783.8 )
SRM (P: -688 Y: 907 ) -> (P: -695.2 Y: 915.0 )
OM1 (P: -533 Y: -524 ) -> (P: -643 Y: -484 )
OM2 (P: 879 Y: 736 ) -> (P: 899 Y: 656 )
OM3 (P: -423.3 Y: 625.9 ) -> (P: -423 Y: 644 )
OMC (P: -358.7 Y: 363.0 ) -> (P: 3641.3 Y: -637.0 )
Koji, Sheila
Koji and I searched around for the source of the periodic (every 4 second) glitch in DARM control when locked on ALS figure attached to 17684. The problem seems to be the DIFF PLL board, we were able to see the glitches in the PLL control signal when it was locked to a marconi instead of the DIFF beatnote. We locked the DIFF beatnote using the COMM PLL board, saw no glitches, and also swapped the ADC cables to make sure the DIFF ADC channels were fine. I don't know where the spare is, so we are elaving the chassis in the rack for now, but the board needs to be swapped before we can get back to locking the full IFO reliably.
Just too be clear, it seems likely these are a different class of glitches from those described in 17576 and linked alogs
We made some additional tests this morning but couldn't see anything obviously wrong. There is a fair amount of RF interference between the 3 VCOs which depending on the exact frequencies looks bad.
Some additional observations:
Eventually we figured out that with DIFF and COMM unlocked, these 4-second glitches were showing up in the Y end green control signal, and not in X.
So we drove down to EY and found that the EY ESD driver was tripped. We could hear that it was trying to reset itself every 4 seconds or so. Untripping the driver made the glitches go away.
It is unclear whether this was actually kicking the test mass, or coupling into the green readout some other way.
model restarts logged for Thu 02/Apr/2015
no restarts reported
model restarts logged for Fri 03/Apr/2015
2015_04_03 10:11 h1iopseih23
2015_04_03 10:12 h1iopseih23
2015_04_03 10:13 h1hpiham2
2015_04_03 10:13 h1hpiham3
2015_04_03 10:13 h1isiham2
2015_04_03 10:13 h1isiham3
2015_04_03 10:46 h1hpiham2
2015_04_03 10:46 h1hpiham3
2015_04_03 10:46 h1iopseih23
2015_04_03 10:46 h1isiham2
2015_04_03 10:46 h1isiham3
2015_04_03 11:20 h1iopseih23
2015_04_03 11:21 h1iopseih23
2015_04_03 11:22 h1hpiham2
2015_04_03 11:22 h1hpiham3
2015_04_03 11:22 h1isiham2
2015_04_03 11:22 h1isiham3
2015_04_03 13:50 h1hpiham2
2015_04_03 13:50 h1hpiham3
2015_04_03 13:50 h1iopseih23
2015_04_03 13:50 h1isiham2
2015_04_03 13:50 h1isiham3
2015_04_03 18:42 h1fw0
one unexpected restart. Multiple restarts of h1seih23 for ADC and PCI issues.
This afternoon Elli, Kiwamu and I phased the AS A 36 in PRX, then again in DRMI. We phased each quadrant to minimize the Q signal with the cavities locked, and the WFS centered, and then adjusted gain to get the same size signal on each quadrant ( the gain adjustments are no more than 40%). This seemed to indicacte that the electronics for each qaudrant are working OK, so nothing needs to be done durring the HAM6 vent. However, after this we tried exciting the angle of the SRM and saw that the signals are still highly coupled between pitch and Yaw
The glitches described in alog 17576 are here again tonight. In addition to the kind of intermittent glitches described in that alog we have verry regular 4 second glitches in DARM control when locked on ALS. This started around 22:38 local time (5:38 UTC on April 3rd). These large glitches cause the half hertz motion of the ETMX suspension. Dan and I are leaving the IFO locked on ALS to see if these go away on their own overnight as the ALS glitches have in the past.
ETMX is swinging in pitch at the half hertz resonance, this is rung up by the glitches. Oplev damping is not an option because the ETMX optical lever laser is dying, as we have known for several weeks now.
Nairwita, Nutsinee
Attached below are the plots of EX CPS sensor vs. EX VEA temperature and EX roof temperature on March 27th and 28th. The more steady line seems to correlate with the roof temperature (out-door) somewhat while the more fluctuated line seems to correlate more with the VEA temperature. We weren't able to get the raw trend of the PEM so the resolution isn't too great. Judging from the data of March 27th CPS vs. VEA temperature, the temperature became steady around 06:00:00 while the bumbling line starting to settle around 08:00:00. Maybe this two-hour delay in response to the temperature can give us a clue of where the source of the bumbling line might be?
There are two lines in ETMX, one of which remains between 50Hz to 100Hz and other one is varying through a large frequency rage (around 10-400Hz). The second line is not visible on 2nd April's spectrogram ( https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=67607 ) of the CPS sensor but it is clear from 1st April's spectrogram ( https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=67615 ) that it actually exists and it's going outside of the frequency range (above 256Hz) and clearly visible in GS13's spectrogram ( https://ldvw.ligo.caltech.edu/ldvw/view?act=getImg&imgId=67616 ). Also to confirm that the the only exists at ETMX, I have done some follow up study on ETMY. There was only one line which could be noticed from 7p.m. on 12th March in the Stage 2 CPS spectrogram and it disappeared around 3a.m. (See the attachment). I did not find two prominent lines like in ETMX. I am looking at the other ISI sensors at ETMY as there is possibility of finding the line at some higher frequency range.
Jim and Dave
we added a new frontend, x1seih23, to the DTS. I moved the HEPI and ISI models for ham2,3 from x1asc to the new FEC.
[Ed. Merilh, Jason Oberling, Doug Cook, Suresh D]
Doug replaced the diode laser at the HAM3 oplev this morning after it was fixed for reducing glitching (SL No. 197) in the optics lab 2. We wanted to let it settle for a while and reach thermal equilitbrium before adjustting the power level. We did that around 12:30PM and I checked the results around 4PM. The laser is still settling down as seen in the attached plots. We plan to monitor it for another day.
Sl. No. 197 Diode laser requires readjustment of power.
Please see the two attached plots.
1) The first shows short term trends of the laser power as obtained from the HAM3_OPLEV_SUM_IN1_DQ channel. The first panel shows the RIN spectrum. Note the two orders of increase in when we go below 1Hz towards 0.1Hz. This indicates power instability at low frequencies (A signature of glitching). The second panel of this attachment shows hte time trend of this signal which shows gradual increase in glitch rate after the first hour or so. The laser is moving from a low glitch rate to a high glitch rate power level due to thermal changes. The third panel shows the same info in greater graphic detail with time evolution of the spectrum (spectrogram)
2) The second attachment is a long term 1s trend of the same SUM signal. It shows that the initial power setting was okay and had few glitches if any. However the power dropped over the following half a day and moved to an unstable zone. It stabilised there and continued to glitch because it has landed at the edge of a stable zone and is now mode hopping.
Cure: Increase power from 47900 SUM counts to 48500 counts. Further one day of observation to see if it has worked.
The diode laser Sl No. 197 which was under observation at HAM3 oplev has been performing well for the past six hours. There were a few minor glitches after Tuesday morning maintenance started. This could have been some heavy stuff moving around on the floor and disturbing the HAM3 Oplev Transciever. The behaviour over six hours has been summarised in the attached plots of amplitude spectrum, time series and spectrogram.
Scott L. Ed P. Chris S. Yesterday the crew cleaned 52 meters of tube and 67 meters today to X-1-8 double doors. Results are posted here. The extraordinary accomplishment of cleaning over 90 meters on 3-31 was very much appreciated, however I have asked the crew to consider a more reliable pace and not burn themselves out. I reminded them that we are only 20% complete with the overall task. Monday will be spent relocating equipment, lights, and support vehicles.
Measurements were carried out on the ETMX effective charge voltage over a period of 3.5 hours this morning. Firstly, some housekeeping, I svn'd up the charge scripts directory to obtain updates from LLO /ligo/svncommon/SusSVN/sus/trunk/QUAD/Common/Scripts/ and persevered local changes. ETMX was selected since it does not currently have low-pass filters installed on each quadrant. It was not necessary to re-align the ETMX optic, since the OpLev was already well centred. Linearization was bypassed (turned off) for the duration of these measurements. The ESD_UL_LL_UR_LR_charge_07-H1.py script was run to drive the ESD at 4 Hz with an amplitude of 130k counts. The script attempts to write a bias to H1:SUS-ETMX_L3_LOCK_BIAS_OFFSET, however, this is nominally turned OFF at LHO. Therefore, it will need to be engaged, and the bias in DAC counts residing in H1:SUS-ETMX_L3_LOCK_INBIAS set to zero. Also, the ramp time should be reduced from 10s to 5s for the duration of the charge measurements. Data was processed using the ESD_UL_LL_UR_LR_analysis_07_H1.m script, charging and charge deviations from today’s measurement can be seen below (all times in UTC). A significant negative effective voltage is observed on the lower quadrants, which are quite variable. Upper quadrants exhibit less charge and are more stable (similar to what was seen for LLO ETMY, before it was discharged) n.b. there is also a large discrepancy between the effective charge reported by OpLev Pitch and Yaw. Previous automated charge measurements carried-out on ETMX by Brett after the most recent vent did not show as larger effective charge (see LHO aLOG entry 16057). Follow-up measurements, may help determine how the charge is evolving. The infrastructure is in place to help determine if any future discharging is successful. Processed results, and analysis scripts have been committed to the sus svn, raw data files have not.
At Jeff K's request, I've included v2 of my charge measurements notes, so as to help ensure the measurement can be repeated.
If the ion pump is open to the chamber you will have significant charge fluctuations over 5 to 10 hours.
Rai -- agreed. This was more to establish that (a) our measurement suite at LHO was functional and blessed by Stuart, (b) to confirm that, "yup! we've still got plenty of charge," and (c) there are locals on site who know how to run the measurement suite. We know that the charge will continue to swing around because, as you say, experience has shown when ion pumps are valved in, and there is some charge on the mass, it varies greatly. As such, we're not going to bother to continue tracking it on a day-to-day basis, or really at all because we know there's nothing we can do about it until we vent and remove the charge as LLO has done.
Today has been a rough day for the input HAM chambers. This morning we found that a channel went dead in expansion chassis (see Richards alog for the resolution), and troubleshooting involved a lot of restarts of SEIH23. This afternoon, after that problem was sorted, Evan found that the SEI MEDM screens for HAM2&3 were frozen. Dave and JimB should be posting a log about the resolution of that issue.
It would be very good if Detchar could at do some comparisons of the H3 IPS on HAM3 HEPI with other sensors to see if the failure that took out this chamber this morning (killing commissioning efforts until ~now) gave us any warning. I attach some dataviewer trends of the IPS blend ins, and I can kind of convince myself that the horizontal loops all look a little noisier 18 hrs ago versus now. Not necessarily true, I haven't looked in any detail and I'm just guessing based on the max/min being noisier 18 hours ago.
Plots like this, for the last couple of weeks. This is a spectra from 23 hours ago before the trip at 01:00 local today. Maybe also look at impacts on the ISI and/or MC2 PR2?
I've attached spectra of H2 and H3 spaced by three hours. There's a clear indication that H3 is going bad on Apr 2 between 18 and 21 UTC. The next spectra pin the start time to between 19:20 and 19:25 UTC (I think that's noon local time). The time series, which starts at 19:15, shows that the problem may come in the form of bursts of noise. Update: I've added an hour-long time series of the channel, high-passed at 10 Hz. The problems start 12 minutes in. It looks like there are bursts of noise, as well as maybe an increase in the overall noise level.
According to the summary pages, this had an impact on the ISI and on the optic motion. The ISI spectrogram and optic motion spectrogram show an increase in noise right around 19:20 UTC. If we need a monitor for this sort of problem, a BLRMS from 30 to 100 Hz would probably work. The sensor is just flat white noise there, until the signal goes bad. Attached is a 12 hour BLRMS showing the onset of the problem.
Sadly, this doesn't show up in the BLRMS signals that we currently have on HEPI. The HPI-HAM3_BLRMS_X_30_100 channel is the BLRMS of the L4C cartesian signal in the 30 to 100 hz band and it doesn't show the sensor going bad. Attached plot is the same 12 hour window that Andy plots just above, and the problem is not apparent. Mo channels, mo problems.
Last night, we again became unable to damp the roll modes (see previous experience in alog 17378) with the usual damping settings. After some random experiments, we became able to damp it with the usal damping settings for some reason. We have no idea why the mode occasionally behave in this way. Note that we use AS_WFS_A for damping them.
(The roll modes)
After the recycling gain study (alog 17645) and ASC study (alog 17646), we fully locked the interferometer with DC readout and 10 W. We immediately noticed that the DARM spectrum was extremely noisy which turned to be high roll modes saturating the OMC DC PDs at the ADC. Looking at the frequency of the peak in the DARM spectrum, we could idenfity the mode -- it was at 13.8 Hz which is the one from ITMX. The peak height was as high as 10-12 m/sqrtHz in the DARM spectrum with 0.1 Hz BW. We went back to ASQ in order to address the issue. The PSL power remained at 10 W. Evan tried different phases (e.g. +-60 deg) and even the negative sign in the damping gain, but none of them seemed to work. This is exactly the same situation as the one previously reported (alog 17378).
(Damping experiments)
- ITMX
Since I knew that it was mostly from ITMX, I first disabled the damping on ETMY in order to make the experiment straightforward. Then I narrowed the pass band on ITMX, which is nominally 1 Hz wide with a center frequency of 13.9 Hz, to 100 mHz passband with a center frequency of 13.8 Hz. They are 4th order butterworth filters. According to foton, this change causes an extra phase rotation of 30 deg, which I did not try to correct as this seemed small enough. Engaging the narrower butterworth, I was able to damp the mode with the same positive gain. This brought the peak height in the DARM spectrum to as low as 10-14 m/sqrtHz.
- ETMY
I then moved onto ETMY to propagate the same modification. ETMY also had a 1 Hz bandpass 4th order butterworth with a center frequency of 13.9 Hz. I tried a 100 mHz passband with a center frequency of 14 Hz. This again caused a 30 deg phase shift, but I neglected it. After engaging the narrower bandpass on both ITMX and ETMY, however the modes slowly started growing up. I tried different phases and negative damping gain on ETMY, but none of them helped. I also tried several configurations -- e.g. disabling ETMY and keeping ITMX, disabling ITMX and running ETMY, and etc. But I did not succeed in damping the modes. Moreover they kept growing on a time scale of a couple minutes.
- Ending up with the same old configuraion
After all, I switched the bandpss filters back to the 1 Hz passband ones to see if I can damp them. Yes, I was able to damp them. The decay time was approximately on a time scale of a couple minutes. No extra phases or sign flips were needed. Unsatisfactory (✖╭╮✖)
not a good idea to use the error signals for damping; using AS WFS requires that the roll to angle TF not have phase shifts at the ~30 deg level.
But the TF includes not only the roll -> angle mechanical TF, but also
(roll -> DC readout -> DARM OLG -> SUS actuators L2A -> WFS) +
(roll -> DC readout -> DARM OLG -> SUS actuators L2L -> WFS L2A)
so its complicated.
But also the seemingly straightforward way of using DARM_OUT that I support has issues since the roll -> DC readout TF changes with beam positioning. But if the spot positions are controlled, this way ought to be best as long as you always have a roll RG in the DARM loop.
Summary: H1:SUS-ETMX_M0_DAMP_L_IN1_DQ looks to have been saturating during the long lock and making glitches in DARM at 9Hz.
In Detchar we're looking at the 2015-04-02 lock from ~8-13 UTC, which had good sensitivity and some amount of undisturbed time. The hveto page for that day has several interesting glitch classes. The first round winner is a series of glitches centered at around 9Hz and associated with the SUS ETM* L1/M1 L channels. These glitches seem to only show up in this high sensitivity lock and not the low sensitivity locks around it (perhaps higher RMS on M0 in low-noise configuration?). Checking the raw data, it appears that H1:SUS-ETMX_M0_DAMP_L_IN1_DQ is saturating.
Note: CIS says this channel is calibrated in um. I don't know whether this is a digital saturation or some physical thing - will consult a SUS expert.
One mystery, to me, is why the glitches occur at GPS times ending in .000, .250, .500, and .750. This might be due to time domain clustering in our glitch algorithm. At first I thought it indicated a digital origin for the glitches, which turns out to have been a red herring.
To follow up on the above, Joe Areeda made omega scans of 50 of these glitches in DARM, a tar file is here. Two examples are attached (each over four different timespans). In these scans, the DARM glitches look like three peaks excited for about a second.
I looked at some of the slow longitudinal channels for ETMX, and it turns out the common tidal control signal for the ETMs was hitting its software limit all throughout Wednesday night.
The first plot attached is a one-hour trend of the ETMX-M0_DAMP_L output, which shows the behavior that Josh found, alongside the common tidal control signal for ETMX (the common tidal signal is the same for both ETMs, so picture this happening for ETMY as well). The DAMP_L channel wasn't saturating, but it was flat-topping whenever the LSC-X_COMM_CTRL signal hit the soft limit at 10 microns. An image of the offending filter bank is also attached.
The 10-micron limit is pretty huge, and we're scratching our heads to figure out how the common tidal drive (which is essentially the low-frequency component of IMC-F) could have acquired such a large DC offset. The third plot is a four-hour trend that shows the offset in IMC-F being offloaded to the tidal as the Guardian state climbs towards low-noise.
The common tidal signal is sent to the L1 stage of both ETMs, where it is combined with whatever other longitudinal drive is beng applied to the optic, and then is offloaded to HEPI. During this lock, ETMY was also being driven by DARM, and the contribution of that signal was enough to provide a smooth control signal to the mass. But ETMX was only getting the common tidal signal, and it was stopping abruptly when it came up against the software limit.
Here's an idea for a control room tool: a script that looks at every CDS filter bank, figures out which ones have their limiter enabled, and checks whether the output is within 10% of the limit value during some span of time.
Jim, Krishna,
After the troubles described in alog 17386, BRS was turned off to allow it to damp down naturally. This morning Jim and I restarted the BRS as attempted earlier in 17309. This time, the amplitude of the balance was low enough and we simply had to reposition the damper and start the software. Everything worked as designed. The attached plot shows the damping of the beam-balance.
After things had settled down I checked the spectra of the signals coming out of BRS and the tilt-subtracted super-sensor and everything looks normal. Wind speeds were at 0-10 mph and the subtraction looked good as seen in the attached pdf. The first plot shows the ground T240 X motion, the BRS_RY ( * w^2/g), ST1 T240 X and the super-sensor. The next plot shows coherences between these sensors.
Better late than never, right?
While I was at EX on this day, I also went out to the chamber and checked the corner 3 CPS rack (re: the bumbling line, most recently discussed in alog 17681) and found nothing amiss. All cables are secured, all CPS field racks are grounded and have all their screws. Haven't had time to look at this in more detail (pulling cables, cards, turning stuff off and on again, etc.).
This seems to just be a problem with the reporting of the status of monitored channels. This shouldn't have any adverse affect on the actual operation of the nodes. And it seems to have been a transient problem, since the issue went away after the nodes were restarted. Still investigating.