Sudden lockloss, no obvious reason in the control room. Looking at the attached plots of the FSS, it seems this started oscillating/became unlocked at the same time as the lockloss. I observed the same thing in a few locklosses this month, see alog 54711.
I believe I found and fixed this, see alog54773.
I saw in Corey's alog this morning that he switched SEI_DIFF to CORNER_DIFF_CPS, I'm guessing because he was having problems and it was a button he could try. It's been in that state ever since, which I think is not the right setting. In the past the FULL_DIFF has caused scattering during high microseism, but I'm looking at data that suggests that is not as much of a problem since we added the R0 tracking on the 14th of this month. Still mining data, so I don't have anything to post right now.
In the meantime, Camilla has transitioned back to FULL_DIFF_CPS at 0:33:32 UTC. This didn't drop us out of OBSERVE, I didn't notice any glitches during the transition.
The surf forecast kind of indicates we may have higher microseism over the next couple of days, so I'd like to ask that we leave SEI_DIFF alone, unless there is very good reason to think it is causing a problem. We could really use a good chunk of time with ~1 micron/s microseism with FULL_DIFF_CPS to see if scattering is still a problem or not.
It's been kind of hard to find segments to compare, with moderate-high microseism, the three attached screens are the best I've found so far. But I think that by looking at the OAF Range integrand blrms, we can say that the R0 tracking has improved the 20-50hz scattering enough that we shouldn't have to change SEI_DIFF, when the microseism is higher.
3 attached plots are 5hr timeseries for different of the 20-29hz & 38-60 range blrms, the ITMY Z .1-.3hz gnd blrms and the R0 L2 drive (indicating if the R0 tracking is on).
First image is from a lock with no R0 tracking and full cps diff on. The top suplot shows a lot of glitching (20-29 hz RLP is high) throughout the 5 hour segment here. Second subplot doesn't show a lot of extra glitching. Third subplot is the microseis for this segment, average value is about 600nm/s.
Second image is for a lock segment with full diff cps on and R0 tracking. Similar microseism, but the top subplot shows a lot less activity. It's subtle, but even the 38-60hz shows less activity with the R0 tracking on.
Third image is with corner diff cps and R0 tracking. The 20-29hz glitching looks about the same as the full diff. R0 tracking was on, similar microseism again. Not a whole lot of difference from the full diff cps with R0 tracking.
I would like some data with higher microseism and FULL_DIFF_CPS on SEI_DIFF, but I think this shows that the R0 tracking has improved the microsesmic scattering.
Sorry about this---I thought I noted somewhere that I made this transition, but looks like I didn't (I blame not being able to get on the network on my laptop last night & then being busy trying to log everything I could for recovery most of the night). I did mention it to Patrick during the hand-off this morning (this node was new to him, and I mentioned the transition, but also that it probably wans't necessary since the microseism was below the 90th percentile. I gave him a brief summary of the node and what it does for us, but that you could also explain it better than I.).
Thanks for catching this!!
QUESTION: We can transition this even when we are in OBSERVING, yes?
TITLE: 01/28 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Observing at 117Mpc INCOMING OPERATOR: Camilla SHIFT SUMMARY: Remained locked entire shift. Out of observing from 19:00 UTC - 20:29 UTC for calibration measurements. LOG: 15:50 UTC Scott tumbleweed chipping on X arm 17:24 UTC S190517H verbal alarm. This is from last year. LLO did not receive this verbal alarm. In the GraceDB log I found: Jan 27, 2020 17:24:28 UTC VINAYA VALSAN Advocate signoff deleted: ADVOK removed and ADVREQ reapplied 18:12 UTC GRB-Long E361379 Fermi Trigger ID 601841483 TRIGGER_DUR: 2.048 [sec] Ignore 18:49 UTC Vanessa to mid X. 18:59 UTC Set INJECT_TRANS to INJECT_SUCCESS. 19:00 UTC Jeff starting calibration measurements. 20:29 UTC Back to NLN. 20:30 UTC Jason to diode room to remove external backup drive from PSL computer. 20:34 UTC Hanford monthly alert phone call received. 20:43 UTC Jason done. 20:44 UTC Back to observing. 21:11 UTC Scott driving down Y arm.
TITLE: 01/28 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 9mph Gusts, 7mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.37 μm/s
QUICK SUMMARY: Locked 9h30.
J. Kissel I've grabbed the full calibration suite today, since (a) we'll take as much data on the sensing function as we cal get, and (b) it's been two weeks since the last actuator measurement set, and (c) This will be a good reference set if we enact ECR E1900049, which we'll have to pull the PUM drivers, replace an hopefully-unrelated-to-the-main-path noise monitor board, and then re-install. We may enact this ECR tomorrow, if we can past the tumbleweeds to get to EX. Processed data will be included in future analysis to be posted later. Total calibration time used: 1.5 hours. IFO was in the now nominal configuration: - 38.0W in to the IMC, with the IFO thermalized for a few hours prior to start of data taking - 50 ct digital offset in SRCL - UIM driver state = 1 (no switchable analog low passes engaged) - PUM driver state = 3 (switchable analog low passes engaged), - ESD driver state = 2 (switchable analog low passes engaged) - L2 OSEM to R0 TOP Coils feedback to maintain distance between reaction chain and test mass chain is now ON. Sensing Function. /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs 2020-01-27_H1_DARM_OLGTF_LF_SS_5to1100Hz_15min.xml 2020-01-27_H1_PCALY2DARMTF_LF_SS_5t1100Hz_10min.xml 2020-01-27_H1_PCALY2DARMTF_BB_3min.xml >> Start time 2020-01-27 19:00:46 UTC Actuation Function. /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOActuationTFs/ 2020-01-27_H1SUSETMX_L1_iEXC2DARM_8min.xml 2020-01-27_H1SUSETMX_L1_PCAL2DARM_5min.xml 2020-01-27_H1SUSETMX_L2_iEXC2DARM_12min.xml 2020-01-27_H1SUSETMX_L2_PCAL2DARM_6min.xml 2020-01-27_H1SUSETMX_L3_iEXC2DARM_12min.xml 2020-01-27_H1SUSETMX_L3_PCAL2DARM_6min.xml Data will be processed in the fullness of time.
Have remained locked. Calibration measurements began at 19:00 UTC and continue.
There is a possibility that we will be burning tumbleweeds. Dates will be announced. LN2 delivery tomorrow is to CP2 and CP3. There will be a site weekly Wednesday.
TITLE: 01/27 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 6mph Gusts, 5mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.40 μm/s
QUICK SUMMARY: Tumbleweed chipping along X arm planned for this morning. No issues at present.
TITLE: 01/27 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
Oooeee, It was a doozie of a shift+. H1 was having issues locking from the middle of the EVE shift, and after seeing repetitivie locklosses at LOWNOISE_ASC, decided to back out some changes to ISC_DRMI (see the diary of grief over the night). Toward the end of the shift, finally made it back to good ole OBSERVING. (Just wish I could have gotten there sooner!)
GC wifi network issue arose at the end of the EVE shift; Richard has remedied this & we are good.
LOG:
Summary: After being down 11hrs, H1 is back to OBSERVING after locking on 1st attempt post-backing out the change to BS/PR3 from Fri.
In hindsight, I probably should have focused on backing out the BS/PR3 changes much sooner, but wanted to make a few attempts to see if I could make it back as is. But with every attempt at locking taking 30-45min each, making a few attempts to give H1 a chance, running an alignment, and then finally putting time into reading alogs/looking at python scripts/trending channels/etc., it just took me alot of time to finally turn things around flying solo in the middle of the night. Overall, I had (3) locklosses at LOWNOISE_ASC (and we ended up having a total of 6 of these LOWNOISE_ASC locklosses over the ~26hrs).
With that said, and as posted, I finally ended up backing out/loading BS/PR3 changes at around 13:55 (5:55amlocal) & then started relocking.
Sure enough, the path looked brighter when DRMI locked on its own on this first lock. About 35min later, H1 was back to NOMINAL LOW NOISE. There were a handful of SDF diffs (BS & PR3 gains which was expected and also some ETMy pit/yaw offsets we've been seeing lately; see attached).
With this being only the first locking attempt after the changes I made to ISC_DRMI.py, I would like a commissioner to just go over the changes I made to see that they look square and good.
Shoot. This was definitely due to the guardian change on Friday. :(
I had assumed that the BS LOCK P gain was set to -1 somewhere, just as the Y gain must be. But, since P and Y for the BS and PR3 have always been treated differently, there must have been another place that I didn't catch and fix.
We ran all weekend without any BS P offloading to the top stage. BS still had normal angular control on, but for pitch it was entirely on the lower stage, and the low frequency pointing wasn't being offloaded up the chain. As long as there were no BS saturations, this shouldn't affect data quality, I believe.
I will continue to look into this, including finding where the gain for P didn't get set.
I'm not seeing an obvious place where we're saturating the BS M2 stage at lownoise_ASC that would be causing these locklosses. However, we certainly should not be running with the M1 pitch gain off. Maybe I'll re-implement the change tomorrow, and can watch the relock after maintenance to ensure things go smoothly.
This is a good reminder of using and posting the SDF diffs. After reverting my guardian code change, Corey had SDF diffs that he accepted that correctly put the gains back to their nominal values, so they were accepted with the incorrect values over the weekend (although the reason they were incorrect was my guardian error). We are continuing to remove unnecessary diffs to reduce 'alarm noise', so that any diffs that do show up should be a cause for a small amount of investigation.
I have made one more modification to the ISC_DRMI guardian, so that it will clear the history of the pitch filter. Since the line 71 was still commented, FM1 was not going to be turned off (which is what used to do the 25 min clearing). The gain was getting set to 0 (which makes the pitch the same as yaw), but the clear history (near line 143, setting the _RSET channel equal to 2) was only being done to yaw. This meant that when the gains were put back to their nominal values near lines 158, the BS would still have all the integrated output. So, I fixed line 143 so that it will clear history for both pitch and yaw. With Corey's addition of the writing the pitch gains at line 158 (this is the piece that I had forgotten on Friday), now the guardian should work as I had intended on Friday, not taking 25 min to clear the BS pitch.
I have saved and svn-ed the guardian, so next time we're out of observing we should load ISC_DRMI. (The reason to not wait until Tuesday is that I think the code as-loaded will cause relocking problems).
Not sure why this change occurred during the re-locking process. Accepting sdf diffs to go into Observe.
Something is rounding these values. Their actual values should be the higher precision ones, but I also don't think that it really matters other than from a consistency standpoint. I'll look into this.
I believe that I found this discrepancy in lscparams.py. It wasn't the code that was rounding it, but it must have been a human. The filter medm only displays the offset values to the precision of 10*-1, even though the actual value of it could be much more. Ex: offset value actual = 10.15, displayed = 10.2
Either way, the place in the code that was doing it I have changed and reloaded the appropriate nodes.
[Corey, Jenne]
We've found that when the BS has integrated up a lot of ASC signal, but then we lose lock, it can take a long time for the flashes on the AS air camera to look symmetric again. While looking at a series of locklosses from TR_CARM, Corey and I found that the let-the-integrator-decay filters in the M1 stage of the BS (which are labeled as 4min) are actually having the pitch output of the BS decay to zero over 25 (!) minutes. Recall that these decay filters are to help with alignment issues due to wire heating, but that wire heating issues should mostly have been ameliorated by the wire baffles. Even if we need these decay filters when we lose lock from NomLowNoise and high power, we shouldn't need them when we're losing lock from a cold IFO (eg. DRMI states). Identical filters with long decay times are installed in PR3, but we don't use that optic for angular control, so it never needs to do any decaying.
The decay filters have names that say 4min, but the poles is at 6.6315e-4 Hz, which gives a ~1500 second decay time. The pole frequency is at a surprisingly precise frequency, which if you include a 2*pi, makes it look like the decay time would be 240 seconds, or 4 min. I suspect that the designer of these filters got mixed up and thought the values in Foton were in units of radians/sec, rather than 1/sec.
I don't know that this is what has been causing problems with locking, but it probably hasn't been helping. I suppose that since we turn on MICH ASC once DRMI is locked, and before we start the carm offset reduction sequence, this residual BS offset shouldn't be a problem for TR_CARM (since it will be taken away by the MICH ASC). It might have been causing problems with FIND IR for DIFF/Yarm, since the Yarm bounces off the BS to get out to its transmission PD.
We should probably change the logic so that if we've lost lock from an early acquisition state, we just clear the BS pit history. We can still use these decay filters for NomLowNoise locklosses, if they seem necessary.
After talking to Sheila and Jim, we all agree that we can probably get rid of these wire heating decay filters, since the new BS baffle was put in pre-O3 to further protect the wires from seeing any heating.
In the first attached figure, we lose lock around 1300 seconds from a TR_CARM lockloss (so no power buildup). Very quickly, we have a transient that get through to the BS_M2_LOCK_P output, and so gets integrated in with the M1 Lock input. But, since we've just lost lock, the ASC output isn't sent to the coils (because the lockloss trigger is stopping the signal), but after ~100 seconds the lockloss trigger is reset in the guardian, so we allow the decaying ASC signal back to the coils, and see a jump in the BS oplev signal. The salient point is that as the coil output filter changes over 20 min, the BS oplev is also changing. If the decay filter were compensating for wire heating, then the oplev should be more constant and flat. Since it's not, we're pushing unneccessarily.
In the second attached figure, we lose lock from a 2 hour long Nominal Low Noise lock around 20 seconds. No major transient in the M2 filter that gets sent to the M1 output, although the lockloss trigger again takes away the coil output for about a minute, during which time the optic swings a bit. But, once the ASC outputs are again sent to the coils, we see the BS move as the coil output is changed. I suspect that we'd be okay here also without having the long decay filters.
Since we would only even potentially need the heating decay filters when losing lock from nominal low noise, it's a bit hard to do chopping tests with and without the filter, so probably we'll just try removing it entirely, and making the pitch filter banks operate the same way the yaws do (without any decay, just normal ramp-down).
I have just changed ISC_DRMI to treat BS and PR3 pitch the same as yaw (as is done for all other suspensions) when clearing the ASC history in the DOWN state. Next time we're out of Observing, we'll reload the ISC_DRMI guardian, so that we can (hopefully) see an improvement in some lock acqusition cases. If BS pointing is a real problem over the weekend, we can revert the DRMI guardian and reload it.
Still to be seen (fair game for an operator to look at over the weekend, otherwise I'll look next week) is whether the MICH BS ASC is in fact undoing / taking care of any leftover decay filter offset when we go through DRMI ASC, or whether this decay filter output has been causing trouble for us during the TR_CARM states. I think that it's probably true that the MICH ASC is doing its job and pushing the BS where it needs to go (we do see improvements in buildups afterall), and that the TR_CARM locklosses are from something different.
Even if the TR_CARM locklosses are unrelated to the BS pitch decay filters, it is still interesting that the TR_CARM locklosses seem to go away so easily after initial alignment. Maybe something to check is where the optics were pointing when attempting and failing TR_CARM earlier in the afternoon before initial alignment, versus where the optics were pointing after initial alignment when we successfully got through TR_CARM.
I have just changed ISC_DRMI to treat BS and PR3 pitch the same as yaw (as is done for all other suspensions) when clearing the ASC history in the DOWN state. Next time we're out of Observing, we'll reload the ISC_DRMI guardian, so that we can (hopefully) see an improvement in some lock acqusition cases. If BS pointing is a real problem over the weekend, we can revert the DRMI guardian and reload it.
Still to be seen (fair game for an operator to look at over the weekend, otherwise I'll look next week) is whether the MICH BS ASC is in fact undoing / taking care of any leftover decay filter offset when we go through DRMI ASC, or whether this decay filter output has been causing trouble for us during the TR_CARM states. I think that it's probably true that the MICH ASC is doing its job and pushing the BS where it needs to go (we do see improvements in buildups afterall), and that the TR_CARM locklosses are from something different.
Even if the TR_CARM locklosses are unrelated to the BS pitch decay filters, it is still interesting that the TR_CARM locklosses seem to go away so easily after initial alignment. Maybe something to check is where the optics were pointing when attempting and failing TR_CARM earlier in the afternoon before initial alignment, versus where the optics were pointing after initial alignment when we successfully got through TR_CARM.
When doing this work on Friday, I had forgotten to ensure that the pitch gains get set back to their nonzero values in the guardian. Corey fixed this, see thread in alog 54746.
We were out of observing for ~40 minutes this afternoon while Daniel and I measured the spectrum of the DCPDs up to 4MHz. The goal of this was to see if it is plausible that noise from the interferometer (laser) around 3MHz is beating with the coherent locking field (at 3.125 MHz) and downconverting to create the noise we see when we have a more power in the coherent locking field. Indeed, there is a lot of noise at high frequencies. Calibrated plots coming soon.
Here is a plot of the data taken from the DCPDs up to 4MHz. I've removed the 20dB of gain and the poles at 265kHz and 290kHz from D1700376 in this plot. This was measured with our usual darm offset, restuling in 10mA on each of the DCPDs (we used one of the single PD outputs from D1700376). We tried to repeat this measurement with a diode set up in the AS air path yesterday, 54636, but the 3.125MHz peak was only 20dB above the noise level, so we weren't able to see any of the other noise apparent in this measurement.
Thanks, this is useful for ongoing CLF work.
For reference, Koji's measurements of the transimpedance are below. You can see that the rolloff flattens a bit in the MHz, so I'm less certain how real the bump is.
https://nodus.ligo.caltech.edu:8081/OMC_Lab/236
https://nodus.ligo.caltech.edu:8081/OMC_Lab/235
does your RF equipment have the ability to make a decently long cross correlation? That is surely what we need if we can conveniently get equipment to do it. Alternatively, you may be able to take spectra simultaneously in a sum and null output configuration, and then subtract them. It won't be as clean as xcorr, but it will show excess in a way that is less succeptible to systematic errors from inverting the PD response. Sum can be picked off from the SQZ chassis, and null just needs the phase shift before an RF splitter used in reverse (depending on the phase convention of the splitter).
Update, I remembered that I had acquired the LISO model and fit it in IIRrational for just these kinds of occasions. These fits should be decent up to 10MHz where the simulation cut off.
scipy ZPK notation:
(array([-1.09367517e+06+9.16935186e+04j, -1.09367517e+06-9.16935186e+04j,
-4.54321270e+01+3.94631525e-01j, -4.54321270e+01-3.94631525e-01j,
-2.85835003e+07+0.00000000e+00j]), array([-1.12584306e+07+1.59926553e+07j, -1.12584306e+07-1.59926553e+07j,
-3.11667017e+07+1.49827313e+08j, -3.11667017e+07-1.49827313e+08j,
-4.41594538e+07+5.59042545e+07j, -4.41594538e+07-5.59042545e+07j,
-1.51046460e+07+2.52263715e+07j, -1.51046460e+07-2.52263715e+07j,
-1.02822684e+05+0.00000000e+00j, -9.54273138e+04+0.00000000e+00j,
-5.08417603e+02+0.00000000e+00j, -4.91824766e+02+0.00000000e+00j]), 2.71124765615596e+56)
foton notation:
ZPK([
-174063.80969071676 + 14593.476738924443*i; -174063.80969071676 - 14593.476738924443*i;
-7.2307475830315395 + 0.06280755795247621*i; -7.2307475830315395 - 0.06280755795247621*i;
-4549205.365087142;
],[
-1791834.8749338891 + 2545310.1382306265*i; -1791834.8749338891 - 2545310.1382306265*i;
-4960334.630691198 + 23845757.522465646*i; -4960334.630691198 - 23845757.522465646*i;
-7028195.359231514 + 8897438.443450466*i; -7028195.359231514 - 8897438.443450466*i;
-2403979.0673290133 + 4014901.7200727463*i; -2403979.0673290133 - 4014901.7200727463*i;
-16364.738450407038; -15187.728699344743;
-80.91717459920993; -78.27634271397135;
], 2.71124765615596e+56, "f")
Attached are the fits and the output of the LISO model. I think the model differs a touch from Koji's measurements on the exact 3MHz cutoff frequency.
Here is a plot (and the data used to make it) of the DCPD output measured with the analyzer in noise mode. The first set of data (in the original log above) were taken in spectrum mode. The dark noise in this new plot was measured durring Friday's EQ in Turkey, the "squeezer blocked" trace was taken durring the commisioning time Thursday (54681). These are calibrated using the filters in D1700376 and Lee's estimate of the OMC DCPD transimpedance above. Both spectra were taken with 30Hz resolution bandwidth and 3Hz video bandwidth. A potential problem with this measurement could be that the squeezer came unlocked and the beam diverter closed partway through the measurement, although that didn't seem to have an impact on the spectrum here.
It does seem suspicous that the quadrature difference, which should represent noise coming from the interferometer light, has a spectrum so similar to the dark noise.
This appears qualitatively a bit different than the post-demodulation measurement of the spectrum at LLO50307. There the blocked and dark noise ASD's changed by a factor of about 1.5. This appears to be substantially less than that at 3.125MHz.
Here is one more plot of the DCPD spectrum up to high frequency, again. In the original version of the attached plot in 54753 I made two mistakes in the calibration, which are fixed here..
The attached plot shows, in addition to the dark noise and locked spectrum, the level of amplitude noise which I think would be needed to create downconverted noise about equal to the current shot noise limited sensitivity. One conclusion is that the dark noise of the DCPDs at 3MHz is too high for us to measure the level of amplitude noise that we need to measure to be sure that we can turn up the CLF power to the level required for the filter cavity controls. However, the difference between the in lock and the dark noise spectrum suggests that the current level of amplitude noise is well above this level, such that it should be dominating the noise in DARM. So something seems to be wrong either with the measurement or with my projections.
One could worry that there might be significant variation between the individual transimpednce amplifiers at 3MHz. We have a measurement made at 3.12MHz with the installed amplifier: 47540 which is roughly consistent with the one that Koji measured and Lee's fit above. Another worry could be that we are running with a different transimpedance than these measurements were taken with, but we are running in high Z which is 400 Ohms at DC according to D060572, My plot of Lee's fit above gives a DC transimpedance of 200 Ohms, but it is ~220 Ohms at 3.125 MHz, so I think it is close enough.