Ed, Patrick 14:50 UTC The IFO lost lock. Guardian reported ISC_LOCK node in error and LOCK_LOSS_SHUTTER_CHECK node in error. The guardian overview turned dark red. I hit LOAD on the ISC_LOCK guardian. Ed came in and I turned around and saw a bunch of frontend models were white. I thought it must have been a power glitch, so I called Richard. He reported that he did not receive any notices related to a power problem and suggested I call Dave. I have done so, but have not been able to reach him yet. I am no longer thinking it is a power glitch. The laser is still up and all of the machines in the MSR appear to be running. I have attached a screenshot of the initial guardian error and the cds overview.
This spectrum was taken with the POP_A PIT QPD offset removed (see snapshots).
Left plot: Current noise agains O1-references.
Right plot: Current noise against tonight's 40W noise, and the noise from last night (POP_A PIT QPD offset was on, TCS ringheater was transitioning - see previous elog.)
Plot 1 shows the DC signals of all 4 I segments of ASA36. Note that seg 3 is ~2.5 times larger than the others.
Plot 2 shows the updated AS_A_RF36_I matrix - the gains for seg 3 have been dropped to -0.4 from -1.
Plot 3 shows the resulting error signal - it now cresses zero where the buildups and couplings for SRCL are good.
Closed the SRC1 PIT and YAW loops with a gain of 10, and input matrix element of 1. I will leave this setting for the night - although it is not in guardian yet.
I accepted the funny matrix in SDF, and added this in the SRM ASC high power state. The loops should only come on for input powers less than 35 Watts. Nutsinee and I tested it once.
Stefan, Terra
We had a large peak rise quickly at 27.41 Hz around 6 UTC. A bit of searching gave us Jeff's alog identifying it as the bounce mode of PR2; as such we lowered gain of MICH from 2 --> 1.2 which eventually allowed it to ring down.
We were wondering whether the auxiliary noise depends on the TCS state or PR3 spot position. Last night we had a ringheater change with several short locks over the night, and tonight we had some alignment change.
Attached are AUX spectra from last night, from early tonight, and just now. For some reason the early tonight spectra were significantly better (albeit not quite O1 quality). We could not correlate it with alignment or heating in a systematic way.
TITLE: 10/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
I would recomend that instead of putting the extra factor of 2 gain in PRCL2, people double the gain in PRCL 1. The guardian doesn't touch the gain in PRCL2, but it will later in the locking sequence adjsut PRCL1 to be the nominal value. If people adjust the gain in PRCL2 and forget to rest it to 1, this can cause problems.
If we consistently are needing higher gain, there is a parameter in LSC params that adjusts the gains for lock acquisition.
The states are designed for in-lock adjustment of the spot position on PR2 and PR3. It is a guardian implementation of the scripts
/ligo/home/controls/sballmer/20160927/pr2spotmove.py
/ligo/home/controls/sballmer/20160927/pr3spotmove.py
So far, turning the PRC1 ASC loops off has to be done manualy.
Instructions
PR3_SPOT_MOVE:
WFS state: disable the PRC1 loop that ordinarily moves the PRM to center POP_A.
Use the PRM alignment sliders to move the beam spot on PR3. The scrips slaves all other optics in order to avoid taxing the WFS.
PR2_SPOT_MOVE:
WFS state: disable the PRC1 loop that ordinarily moves the PRM to center POP_A
Use the PR3 alignment sliders to move the beam spot on PR2. The scrips slaves all other optics in order to avoid taxing the WFS.
(see also alogs 30030, 28442,28627)
Stefan, Sheila, Terra, Ed and Nutsinee
We have found that alingment and TCS together can improve our noise hump from 100Hz-1kHz. We have reverted both alignment and TCS changes to July, and we seem to be stable at 50Watts with a carrier recycling gain around 28.
TImes:
22:18:18 Oct 23rd (before alingment move, at 40Watts) and 22:28:12 (after)
ten minutes of data starting at 23:33:33 is with the better alingment at 40 Watts, 10 minutes starting at 0:19 UTC Oct 23 is at 50 Watts. (we redid A2L at 40 Watts, but not 50 Watts, there is MICH and SRCL FF retuning still to be done.)
We saw that the TCS changes of the last few days made a small improvement in the broadband noise lump from 200Hz -1kHz, so we decided to retry several of the noise test we had done before without sucsess. We only moved the POPA spot position in PItch, moving it in yaw made the carrier recycling gain drop but didn't help the noise. The attached screenshot shows the spectra, and the coherence with IMC WFS, our best jitter sensors in lock. We have lots of coherence with these signals, at frequencies above where the HPO changed the spectra.
Apparently this alog was not clear enough. Kiwmau recomended bigger fonts.
The main message: The blue trace in the attachment was taken at 50Watts, and there is no broad noise lump, just the jitter peaks from structures on the PSL
My earlier comment assumption was correct: during the settling of the recent ring heater changes, Mode26 had shifted outside of the usual guardian-controlled bandpasses; an already existing filter needed to be turned on. After this, we powered up and damped with no problems. Locked for 2.5 hours at this point needing only normal phase changes. I walked Ed through how to check on this and change the filter, but this large of change only occurs after a ring heater change so will not be a normal issue operators need to worry about.
Reminder that I've requested to be called anytime there is a persistant PI problem.
I've added step-by-step instructions for how to attempt to handle this scenerio in the Operator PI Wiki (under 'If a PI seems unresponsive') if for some reason I can't be reached. PI Wiki can also be opened from PI medm screen. Working on automating this.
I arrived to the PI experience that Patrick was having and tried my hand at damping to no avail. This seems to have been plaguing his shift. Diag main was complaining that Nominal ring heater settings for ETMY was 'Off". I read Sheila's aLog about turning them on but trending back to a time when we were having somewhat decent lock stretches (around 10/10) I decided to try turning the ETMY heaters off and see what hatches. Repeating the same process that Patrick was dealing with didn't seem thatit would get me any further than he had gotten. I'll be consulting commissioners soon on this matter.
Terra called. I put the ring heater setttings back and she and a few others will be on site later. She also gave me some valuable info on the Mode26 BP filter settings that include switching ON FM1 and switching OFF FM3.
I kept losing lock from NLN at 26 W after a fairly constant amount of time. Each time ETMY would saturate and the whole DARM spectrum would jump up for a while before hand. I have begun to suspect that it might be related to a fairly broad hump seen in Terra's PI DTT template that grows in height and width as the lock progresses. It is centered somewhere around 15008 Hz and the peak grows from around 100 to 10000 before the lock loss. I tried sitting at DC_READOUT and then INCREASE_POWER and playing around with a bunch of PI settings, assuming it was related to either mode 18 or mode 26. I tried various filters in the H1SUSPROCPI_PI_PROC_COMPUTE_MODE26_BP filter bank, different phases and gains and even turned off and back on the entire PI output to ETMY (H1:SUS-ETMY_PI_ESD_DRIVER_PI_DAMP_SWITCH). Nothing really seemed to have any effect and it eventually broke lock again. Relocking has been fairly robust. The IR transmission at CHECK_IR has been kind of poor towards the end of the shift, but it doesn't seem to hinder further locking. At the beginning of the last lock at NLN I changed the TCS ITMX CO2 power by a small amount per Sheila's request. 07:40 UTC NLN at 26 W 07:50 UTC Damped PI mode 27 by flipping sign of gain 07:56 UTC Played around with damping PI mode 18 and 26. Not sure if they just came down on their own. 08:10 UTC Damped PI mode 27 by flipping sign of gain 09:05 UTC Lock loss. EY constantly saturating and PI modes 9, 18 and 26 ringing up. Attempts to damp these by changing their phases did not seem to help. 09:30 UTC Lock loss immediately upon reaching DC_READOUT. IFO got that far without intervention. OMC SUS and HAM6 ISI tripped. 09:35 UTC X arm IR transmission is ~ .4 at CHECK_IR. Able to bring it to ~ .7 by adjusting H1:ALS-C_COMM_VCO_CONTROLS_SETFREQUENCYOFFSET from 0 to ~ -156. This dropped Y arm transmission. Brought back by moving H1:ALS-C_DIFF_PLL_CTRL_OFFSET. 10:00 UTC Pausing at ROLL_MODE_DAMPING. Waiting for ITM roll modes to damp. 10:12 UTC DC_READOUT_TRANSITION. Stable here. 10:13 UTC DC_READOUT. 10:17 UTC Stable at DC_READOUT. Moving on. 10:25 UTC NLN at 26 W 10:35 UTC Flipped sign of PI mode 27 for third time since NLN. 10:57 UTC Flipped sign of PI mode 27 again 11:18:50 UTC Changed TCS ITMX CO2 power using rotation stage. Changed requested power from 0.200 W to 0.201 W. 0.195 W measured out at 42.2733 deg changed to 0.195 - 0.196 W measured out at 42.3040 deg. 11:24 UTC EY saturating again 11:25 UTC Tried dropping power to 10 W by going to manual -> adjust power -> auto, power request through guardian. Lock loss. Peak on PI DTT at 15008.9 got really broad. HAM6 ISI and OMC SUS tripped. 11:45 UTC Having hard time on CHECK_IR getting stable transmission. Moving on. 12:06 UTC Pausing at DC_READOUT. 12:12 UTC Stable. Moving on. Various large optics saturating until reaching ~ NOISE_TUNINGS. 12:20 UTC NLN at 25.9 W 12:22 UTC Changed TCS ITMX CO2 power using rotation stage: req .200 W -> .198 W meas .196 W -> .194 W meas 42.2800 deg -> 42.2483 deg 12:30 UTC Flipped sign of PI mode 27 gain 12:46 UTC Flipped sign of PI mode 27 gain 13:18 UTC Lock loss. 13:46 UTC Pausing at DC_READOUT. Played around with various settings for mode 18 and 26. 15008 Hz line seemed to come down a little, but not sure it was anything I did. 14:12 UTC Going to INCREASE_POWER. Sitting at INCREASE_POWER trying lots of different things to damp mode 26. No effect.
I believe this was all due to a different BP filter on Mode26 needing to be engaged (one outside those that are currently controlled by guardian) in light of frequency shift due to recent RH changes. I have called and told Ed which to turn on. I repeat: please call me at anytime if there is persistant PI trouble. Often a simple fix might save a night of not locking/stress and I'm happy to be woken up. PI Help Desk phone number is on control room whiteboard.
TITLE: 10/21 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Flipped the gain sign and added -60deg to 4735Hz damping filter (added to Guardian). The mode hasn't caused us any trouble tonight (as mentioned in alog30713). Lockloss multiple times at DC Readout. I found that sitting at DC_READOUT_TRANSITION for a while helps (unclear to me why -- all the control signal looks fine). Had trouble locking FSS many times tonight. Every unsuccessful FSS lock seems to cause the noise eater to complain (and also kills ISS and PMC). I ended up locking FSS in the LVEA near the PSL where I can easily toggle noise eater. A successful trick seems to be locking FSS first, and the rest (ISS, PMC) will follow. Just keep hitting ON button on the auto lock until it stays on. Had no problem locking PRMI/DRMI last time I tried (besides alignment issue). Watch out for PI Mode27. The damping phase changes constantly.
- Guardian is now able to go to low noise at any power (see alog 30730)
- While the ISS isn't touched in the Guardian, its gain needs to be set for the final power:
25W: H1:PSL-ISS_SECONDLOOP_GAIN = 13dB
50W: H1:PSL-ISS_SECONDLOOP_GAIN = 7dB
- At 25W I did a quick SRCFF tuning (no filter - just gain to -0.55) (now in guardian). This produced a broad notch in the SCR coupling around 80ishHz. Attached is a residual coherence plot.
- Next I tried a number of things to attached the noise below 100Hz:
- Moved both compensation plates by 100urad in PIT and YAW. Moving them fast clearly produces scatter shelfs, but there was no obvious noide difference when the CPs were parked at a new location.
- Turned off the ESD bias on ETMX, ITMX and ITMY (no obvious effect)
- Added 20000cts of ESD EY offsets in pringle-shape to avoid zero-crossing (no obvious effect)
- Noticed that ITM L3 STATE REQUEST was 1 (LP off) - I am not sure the switch does anything on the ITMs, but I changed it to 2 (LP on) (no effect)
- Slightly lowered the HARD ASC loop bandwiths. The shelf just below 25 Hz is from the ASC and scales accordingly.
- Temporarily tunrned off the 40ish Hz cal lines to make sure they don't introduce non-linear noise (they don't).
- While I didn't close the SCR1_Y loop, I did look at the AS_A_RF36_I_YAW signal - it looked like a perfectly fine signal and we could use it again at 25W.
- Just remembered that we are still on the POP_X WFS (with POP beam diverter open). We should switch back to REFL.
Attached is a noise spectrum. Sensmon claims 60Mpc, but the 331Hz peak is actually 8% high, so I guess we are closer to 65Mpc.
Summary:
There is a broad peak in the DARM spectrum around 5kHz which is consistent with the TEM10 being rung up at that frequency.
Details:
The attached MATLAB file produces an animation that shows the DARM spectrum around 4700 - 5500 Hz. It updates this spectrum for 50s intervals around the time that the IFO is locked. Very quickly we see a broad peak (around 50Hz wide) form in DARM around 4900 Hz. This peak moves up in frequency as the lock is maintained.
I have the online simulation of the HOM spacing (frequency vs time) shown in the top plot. As the anmation steps forward in time, we can start to find the maxima of the broad peak in DARM. I've taken the location of these maxima and plotted them in the FREQ vs TIME plot (I've only picked the location of the maximum value of the broad peak and tried to exclude all times before the lock is acquired and the frequency starts to change). The transient behaviour is quite consistent with what the simulation predicts.
The two figures correspond to two recent locks but different ETMY ring heater settings:
After the ETMY ring heater is turned on and stabilizes, we expect the HOM frequency to shift down by about 60Hz, according to the simulation. We see a commensurate change in the location of the peak in DARM.
If this is the TEM10 mode in the arm, then what is ringing it up?
Time 2:
More to come ...
(Download and run the MATLAB file to see the animation).
TEM10 mode overlaps with 47.5 kHz mechanical modes (see here) which we see aliased down around 18050 Hz. We have two PI in ETMY (47495 Hz and 47477 Hz) which ring up strongly during the beginning of the lock (through the first hour), which would of course ring up TEM10 in turn. 47495 Hz broke the second lock time you look at after about an hour. I'll run your matlab file over the weekend and compare to the amplitude behavior of these PI. (Also note that the ETMY ring heater change you mention was specifically to move into a zone where there's more overlap with these mechanical modes so that I could check our ESD damping capabilities.)