TITLE: 08/06 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Calibration
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.04 μm/s
QUICK SUMMARY:
!4:53 Re-Running PEM injection script as it may have been terminated prematurely.
TITLE: 08/06 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ed
SHIFT SUMMARY:
Ran Alignment by hand (i.e. bypassing the automated INIT_ALIGN alignment guardian node).
Needed to pitch PRM quite a bit during alignment (and accordinly a bit of pitch with the BS).
Have FM1 (-20dB) still ENABLED for the PRC1_P filter bank. (This was to allow one to get past DRMI in locking)
Observed broadband noise on DARM. Andy L noted a possible suspect for this is the Squeezer.
LOG:
During the current lock and after being at NOMINAL LOW NOISE on the order of 45 min, all of a sudden have begun to see a broadband bump on DARM (attached is a screenshot showing instance where the bump is seen on the DARM spectrum & you can also see how often it is occurring on the DARM BLRMS striptool). It is also seen on DMT Omega. The range has only taken a small hit of about 5Mpc (down to 112 Mpc from 117Mpc).
Only difference I accepted for this lock was the enabled filter for ASC PRC1_P FM1 (-20dB).
Will start taking a look at the H1 Summary Pages to see if I can glean anything from them.
Note:
The DCPD sum-to-null ratio calculated as a squeezing diagnostic jumps up in these bands at the time of the noise, then goes back down. Attached is a timeseries, with the excess noise period from 12:14 to 12:37. Also a spectrum of DCPD sum and null. This seems like it could be an error in squeezing angle. There was also a half-hour period yesterday at 1:30 UTC where the same kind of noise happened. Jake Slutsky suggests that this could be tested by shuttering the squeezer. If the noise comes back, you could try going out of observing and closing the shutter for a few minutes (if this wouldn't break anything).
Thank you for the quick response, Andy! (I tried looking at the Summary Pages, but many didn't have recent data posted (I did see this noise for IMC Freq Noise...see attachment #1).)
Attachment #2 is a plot showing how this period of noise has stopped *knock on wood*.
What To Do When It Returns
I've actually not had experience with closing a Squeezer shutter (guardian generally handles all things Squeezer for us). But after snooping around, I see that there is a "SQZT6 CLF" Shutter on our Shutter Summary medm window (see attachment#3 for a screenshot showing this shutter). I wonder if this is the shutter one would use to close a Squeezer Shutter when investigating this noise.
After checking many more things, I find that there's a lot of noise in the FSS. Attached is a spectrogram of FSS fast, and spectra of that and the FSS mixer channel. There's a much smaller increase in ISS noise. This seems more likely to be the cause that either the IMC or the squeezer, since it's upstream of both of them. So I would suggest checking the health of the FSS loops (and maybe nearby things like PMC) before trying anything else. That's assuming this noise returns.
Seems like something was going on out of the band in FSS (look at PC MON, yellow) though we cannot say if it was from FSS servo itself or laser.
Noise eater (H1:PSL-MIS_NPRO_RRO_OUTPUT) was within its nominal range of -5852+-50, ISS was OK, these two are exonerated.
Tidal was not railing anywhere.
Next time this happens, please run the dtt template here: /ligo/home/keita.kawabe/Templates/FSS_highfreq.xml
This looks at IOP channels, we can only see things up to ~30kHz so the chances of seeing something is not that high but it's better than nothing.
@DetChar & TJ Massinger -- can you plot the same spectrograms of PSL-FSS_MIXER_OUT_DQ and IMC-F as you did for LHO aLOG 51260? Leading question: are these two aLOGs (51260 and 51053) documenting a similar noise problem?
Summary of Down Time during shift:
Locking notes: (Alignment notes were posted earlier.)
Thanks Corey! Hopefully things are now guardian-ified so that no one needs to do anything by hand for either PRX in initial alignment, or DRMI ASC. I turned off (and accepted as off) the PRC1 P FM1, so that it is back to how it has been. However, since we don't use PRC1 during full IFO anymore (we use the ADS system to control the PRM pointing in full lock), it would have been okay to turn off this filter :)
Jenne: Ahhhh, thanks for clarifying. (I think I noticed the Input was OFF, but can't remember.) Wonder what got me through DRMI last night on Lock #3...Lock#1 & #2 had locklosses at this point...but maybe it was waiting for ASC signals to converge that helped? But I remember waiting for Locks #1 & #2. So maybe just luck of H1 was looking over me or it was a nice IFO Voodoo step which helped me get through in the middle of the night. ;) Anyway, thanks for the note & glad recovery was smooth today!
Upon returning to locking, the first oddity to note is an odd ALSx.
The green X-arm locked up fine, but it had a visible oscillation which showed up fairly big, and then became smaller (see 1st attachment).
Decided to stay at LOCKING_ARMS_GREEN for a while (well, actually I couldn't move on Thinking something was swinging X-arm, I scanned SEI & SUS screens and did not see anything notably amiss. It seemed to go away, but after about 10min, ALSx seemed to get noisy again (see 4th attachement).
Moved on and hoping for the best.
Since Cheryl mentioned there were issues with running an alignment at the PRC step, decided to endeavor to run an alignment "by-hand" (i.e. the way we used to do it before the new INIT_ALIGN guardian automated most of the alignment.
Not totally sure how to detach the new automated aligning guardian, INIT_ALIGN from alignment. Tried taking it Manually to IDLE, but When I selected INITIAL ALIGNMENT on ISC LOCK the INIT_ALIGN guardian looked to jump into action. So next up I took INIT_ALIGN to an OP state of STOP, which moved to PAUSE (one of these steps turns the node YELLOW), and this looks liked it allowed me to keep running a manual alignment.
0) 8:01utc: ISC_LOCK to INITIAL_ALIGNMENT
1) Green Arms: Aligned these normally and with no issues
2) Input Align: Offloaded normally & completed.
3) PRC: Walked through each step since there were issues with this earlier. Notes below.
4) MICH_BRIGHT_OFFLOADED---went to DOWN and stayed there...so moved to....
5) The old MICH_DARK_ALIGN
6) For kicks, tried MICH_BRIGHT_OFFLOADED (because maybe it was not in a closer alignment for it to work this time)---Guardian completed this time!--->OK
7) SRC--->SRC_ALIGN_OFFLOADED----Guardian completed these steps on its own---> OK
8) 9:53utc Alignment COMPLETE (ISC_LOCK taken to DOWN): At this point, this very long by-hand alignment was complete! Returned INIT_ALIGN to EXEC
NOTE: I did not make any change to lscparams.py file. So that PRXY gain is still at the low value of -5200 (atleast for my by-hand alignment).
TITLE: 08/06 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: relocked with much intervention, lockloss due to the Montana EQ, many issues relocking, handed off to Corey
LOG:
Attachments:
TITLE: 08/06 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
Wind: 5mph Gusts, 3mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.04 μm/s
QUICK SUMMARY:
Cheryl was recovering from a Montana EQ when I walked in, but looks like we still have issues which are tied to previous locking issues before her shift.
Cheryl mentioned there were no trips or other obvious bad things from the EQ, but she did mention issues with the alignment, so she went for an Initial Alignment, but there have been issues at the PRC alignment step. (and I still need to read the alogs from Jenne & crew regarding PR issues from previous locking attempts).
Since this looks like a non-standard issue, I switched OBSERVATORY MODE to the CORRECTIVE MAINTENANCE state at7:11utc to mark down time.
Lockloss: M 4.1 - 4km NW of Manhattan, Montana
[Cheryl, Jenne]
We're back in Observe.
We were a little fooled by DRMI ASC today, thinking that we needed to redo initial alignment a few times - we didn't. The problem seems to be that with the new alignment, the POP_A QPD is quite close to the edge (0.8ish in pit) when we have DRMI locked, even if we've just done a fresh alignment (or three...). When PRC1 comes on with that large of an error signal and its nominal gain, it pulls the PRM quite hard over and the rest of the ASC fails to follow along. What ended up working this acquisition was turning on the FM1 -20dB filter for PRC1 pit and having a much longer DRMI ASC convergence time.
I checked that for the 30 hour lock this had been true, and we must have just squeaked by because the ASC didn't cause a lockloss. I haven't checked times from before last Tuesday, but I suspect that the POP QPD may not have been as close to the edge before we arrived at our new alignment. Or, something in the initial alignment is leaving the PRM farther away from its DRMI ASC position than it used to.
WP 8302
I wanted to get some calibrated spectra of the MICH/PRCL/SRCL degrees of freedom and noticed that the DRMI FOM - i.e. the plot on nuc 4 was not calibrated correctly. The first thing to fix was the outdated actuator filters in the CAL-CS model (the no longer matched the filters that were engaged for BS/PRM/SRM).
While Jenne and Cheryl were relocking today I switched the filters to what I think they will be (will confirm once we are fully locked again).
This has caused several SDF diffs in CALCS which I have already accepted in the safe snap but will need to be accepted before we go to observe.
Apparently they are not monitored in Observe, because once we got to NLN, there were no SDF diffs.
TITLE: 08/06 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
Wind: 11mph Gusts, 7mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.04 μm/s
QUICK SUMMARY: Jenne helping with the relocking, delay in alog due to relocking issues, beginning an initial alignment
Cheryl and I are doing a quick re-initial alignment starting with XARM IR, since the DRMI ASC keeps pulling us out of lock. (I'm hopeful that we'll get away with just doing this ASC, and not the green arms). While watching, I was curious why we seemed to be *railing* the PRM during PRX lock. The PRCL gain was set to -5250000, which seems more like a typo than a gain that usually goes in our LSC dofs. Looking through the svn archive, this change was made between the April 5th 2019 checkin and the April 23rd checkin of lscparams. There are no notes either in the inline comments or the svn log as to why this would be, since the previous value was -3200.
I changed the params value to be -5200, but by then we were moved on to other parts of init align. If Ed finds a problem with this state when doing initial alignment tomorrow, we can think about taking a quick open loop gain measurement.
UPDATE: We did an initial alignment starting from green arms, since we weren't able to get past TR CARM. PRX was unable to lock with the gain value of -5200, but was able to lock with -5200000 (-5.2e6), but this value railed PRM. So, I tried -5.2e5, and PRX was locked and PRM was not railed. So, we'll try this value for a while, but again if there are problems with the next initial alignment, someone could revert this value in lscparams.py.
Since this looks like a possible long night, want to make a note of where the value for this gain is (since I'm a novice with editing guardian scripts):
File:
/opt/rtcds/userapps/release/isc/h1/guardian/lscparams.py
Value in question:
'PRXY' (line #81 as of tonight) with current giganto value of -520000 (-5.2e5), BUT previously it was a larger -5250000 (-5.25e6).
My Note For Night:
Will be going through an alignment soon. And will try to do so WITHOUT the new INIT_ALIGN guardian node trying to step through everything automatically. But if I have issues, I will revert to the LARGER issue noted above.
TITLE: 08/05 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Currently recovering from a commissioning lock loss, ending a 33 hour lock. I ran initial alignment because DRMI and PRMI looked awful. During the IA, I had to move PRM tens of urads in pitch to get PRC to lock.
LOG:
On Monday evening, Koji and I went out and measured the shot and dark noise of the IMC/CARM PDs. Then, Koji taught me how to think about photodiode noise. Slides 35 and 36 of G1401145 Model:where nV is the voltage noise floor, T is the transimpedance, e is the electron charge, R is the responsivity of e λ/(h c) = 0.858 A/W, QE is the quantum efficiency of 0.9, PDC is the DC power from the incident light, and Pdet is the power apparent from the intrinsic detector noise (i.e. the incident power where shot and dark noise are equivalent). We fit the measured shot and dark noise spectra for IMC REFL I/Q, REFL_A 9I/Q, REFL_B 9I/Q, and at the CARM and IMC servo board OUT2 locations with the nominal gain slider settings (OUT2 connected to IN1, CMB IN1GAIN = 6 dB, IMC Board IN1Gain = 28 dB). REFL_A spectra are plotted in attachment 3 as an example. For dark noise I shuttered the PSL. For shot noise I measured with 1W, 2W, and 4W input requested power, and recorded how much light was reported by the LF channels:
Light Level REFL_A_LF [mW] REFL_B_LF [mW] IMC_REFL_LF [mW] ---------------------------------------------------------------------- Dark 0 0 0 1W PSL 3.51 3.24 1.87 2W PSL 7.38 6.80 3.92 4W PSL 15.08 13.90 8.01 Full Lock 4.36 4.02 1.15Then, I found the average noise floors for each spectrum between 3 and 10 kHz, and fit the model to the data. (attachment two) I have let the transimpedance number from the fits capture the servo board gain sliders.Photodiode Transimpedance [V/A] Pdet [mW] IN1 Gain Slider [dB] ------------------------------------------------------------------------------------- CMBoard 730.4 2.2 6 IMCBoard 2357.7 3.5 28 IMC_REFL_I 128.6 1.6 IMC_REFL_Q 127.5 1.7 REFL_A_9I 154.9 2.0 REFL_A_9Q 151.8 2.3 REFL_B_9I 152.4 2.1 REFL_B_9Q 152.2 2.1
The transimpedance numbers above are incorrect. In the model I did not convert mW to W for the calculation, so all numbers were scaled down. Posted is a corrected plot.
Comments from Daniel "... the transimpedance typically describes the electronics gain. However, yours doesn't. The shot noise eq is missing a sqrt(2), and the demod will fold noise below the LO and above the LO on top of each other, which results in another sqrt(2). So, your T is twice as large as the electroncis gain. For IMC REFL the mesured RF gain of the PD is 353 Ohm, the demod gain is ~5.4 which gives 1.9k." Essentially, the above model ignores a overall factor of 2, and collects the demod gain of 5.4 into the transimpedance, which is not correct. So, for IMC_REFL_I above, the transimpedance is actually 4065 / 5.4 / 2 = 376 ohms, close to the direct measured 353 ohms Daniel reports. REFL_A_I transimpedance: 4903 / 5.4 / 2 = 454 ohm REFL_B_I transimpedance: 4814 / 5.4 / 2 = 446 ohm
15:01 PEM injections are done.