There were no sign of anything going unstable. It's just dropped. I tried engaing the new PRC1 filter that Sheila removed from the Guardian as IFO goes to high power but it's still causing instability at that point.
Tonight (Nov 3rd around 8:20 UTC) we saw something none of us had seen before, ETMY violins rang up very suddenly. The noise is DARM was extremely high, and we saw beat notes between violins at low frequencies in DARM once things calmed down.
Nutsinee has been damping them very effectively, but we don't know why they rang up so sudddenly
I have made scripts that we can run that will adjust the a2l gains for PR2 and PR3. These use the same a2l engine that the test mass script does, it just sets the settings for PR2 and PR3. They can be found in the same location (just checked in): .../userapps/isc/common/scripts/decoup/a2l_min_PR*.py, where * is either 2 or 3.
Each script has been run tonight, so we now have non-zero a2l elements for both those mirrors.
Nutsinne is having trouble relocking, with instabilities around 3.5 Hz that ring up when she engages PRC1 ASC. We removed both the new offset from POP A pit and the new a2l from PR3, hoping that this will help.
Stefan's offset for POP A was -0.05, Jenne's a2l coefficients were PR3_M3_DRIVEALIGN P2L 0.85 and Y2L 1.85
The problem was the new cut offs added to the PRC1 loop tonight, they seemed fine at low noise but aren't stable when the loops forst come on. We have just removed them from the gaurdain for now.
Hmmm, odd. Anyhow, the cutoffs now come on in LowNoiseASC rather than earlier.
I tried to set up an acoustic injection that reproduced several of the bumps in DARM (last time I only did one shaker bump) and tried PR2 pitch, yaw and length injections and IM2 pitch. The best was PR2 pitch, see attached figure. The behavior at other bumps was similar.
x5 scatter reduction at 50Hz.
=======================
Taking Robert's observation as a hit, we moved the PR2 spot position in PIT (using the PR2_SPOT_MOVE state), looking for variations in scatter coupling while driving with acoustic noise.
Indeed, as we move from -275urad to -302urad PR3 PIT alignment counts, the scatter peak at 50Hz varies like a bar code reader.
The starting alignment (PR3 PIT = -296.72urad) corresponds to a maximum in scatter at 50Hz . We found minima at -301.72, -299.32, -292.47, -290.22 and -278.52 urad or PR3.
We thus picked -292.47, which seemed to be the best perfoming spot. It corresponds to a PR2 PIT alignment of 1767.24urad (see alignment snap shot).
The attached plot 1 shows the undriven DARM noise (blue), the acoustically driven DARM noise at the original position (-296.72urad of PR3 PIT), and the acoustically driven DARM noise at the final position (-292.47urad).
Plot 2 is an alignment snap shot.
Finally, to lock this position in, we had to add a POP_A_PIT_OFFSET of -0.05. Jenne ran a PR2 A2L script (demodulating PRCL) to mark the new PR2 spot position. The required M3 drive align numbers are in snapshot 3.
I ran the current version of the calibration pipeline over a stretch of O1 data to reproduce the kappas and compare to those in the C02 frames. The filters file used was aligocalibration/trunk/Runs/O1/GDSFilters/H1DCS_1131419668.npz, as suggested by the calibration configuration page for O1: https://wiki.ligo.org/viewauth/Calibration/GDSCalibrationConfigurationsO1#LHO_AN2 The agreement looks quite good. Time series plots of the kappas and the cavity pole are attached. The start time used here was 2016-10-04 12:41:19 UTC (GPS 1127997696).
Here is an updated plot of the same time interval (approximately) with the SLM tool kappas included. Interestingly, kappa_c and the cavity pole are in good agreement for this O1 stretch of data, while kappa_tst and kappa_pu are not.
alog 30897 has links to SRCL, PRCL and MICH BruCo's. Here hare some interesting highlights from those graphs:
SRCL:
Sure looks like our SRCL sensing is limited by input YAW jitter below maybe 25Hz.
BTW: MICH sees the same noise: CAL-CS_MICH_DQ
PRCL:
This looks like all our PRCL noise is jitter in the recycling cavity...
( ASC-POP_A_NSUM_OUT_DQ ... PIT is more than SUM ...
ASC-POP_A_YAW_OUT_DQ ... or YAW. )
Also interesting:
ISI-HAM3_SUSPOINT_MC2_EUL_L_DQ
I attached all plots for archiving purposes.
Out of curiosity, I looked at coherences of channels listed above from 24Oct2016 (Stefan's data above), and 24Jan2016 (a time when Gabriele ran a BruCo, so was probably a good lock time).
In the attached screenshot, the right half is coherences, and the left half is spectra of those channels. On the coherences half, top right is SRCL vs. IMC-MC2_YAW (ASC control signal), bottom right is SRCL vs. WFS_B_Pit (ASC error signal), top left is PRCL vs. POP_PIT and bottom left is PRCL vs. POP_SUM. On the spectra half of the screenshot, the top row is the length DOF signals, and the lower 2/3 of the plots are in the same locations as the coherences plots. In all plots, green is now-ish and blue is O1 config.
[MattE, Jenne]
We put in modification filters to the IMC WFS loops to move the existing lowpass filters from ~8Hz down to ~1Hz. Both filters (FM9 and FM10 in each DoF) must be on to actually get the full lowpassing.
This didn't change the SRCL noise, but it did reduce the coherenece with the MC2 angular control signal. In the attached screenshot, red is from earlier today (note that it is much less coherent than the green traces from 24Oct2016), and brown is after turning on these new lower lowpasses. I removed the blue and green traces from earlier to avoid too many traces on the plot, but the brown now is basically just as good as the blue from January.
Ed, Patrick and Jenne
Apologies if this is a repeat, but I've never seen this and was at a noticeable loss for action to take.
After the Lockloss at 23:30UTC, there was a diag main message that indicatd that ISC_LOCK wasn't going anywhere until the Fast Shuttter was tested. I'm making a Wiki in Troubleshooting H1 to address what to do in this situation. Jenne came to the rescue. Thanks Jenne!
Took well over 60 minutes to overfill CP3 by doubling LLCV to 36% open first, then open a little bit more, then a bit more, until finally it overfilled, I will change the setting LLCV from 18 to 20%.
Matt E., Dave, Jenne, Jeff K. and I tracked the problems with PI down to settings not being restored or not being restored correctly by SDF after the reboot on Tuesday. It is unclear if the problem with SDF was human or computer error. Jeff K. and Matt reverted the PI settings to those from a week ago. It appears that mode 27 is now back under control. 15:00 UTC Robert to end X to perform injections while at NLN 15:52 UTC Lock loss (PI mode 27). Tried changing sign of gain and turning off damping to no effect. 15:52 UTC Christine and Karen to mid stations 15:54 UTC Kyle to end X 15:54 UTC Robert back 15:56 UTC Robert to end Y to pickup hardware 16:15 UTC Changed to VERY_WINDY_NO_BRSXY ? Robert back 16:19 UTC Pausing at violin mode damping 2 waiting for roll modes to damp. 16:33 UTC Kyle back. Checked that RGA does not have a leaky valve. It does not. 16:50 UTC Karen leaving mid Y 16:52 UTC mode 27 starting to rise 16:59 UTC Lockloss Changed to WINDY during CHECK_IR 17:46 UTC Stopping at VIOLIN MODE DAMPING 2 again. 17:59 UTC King soft water through gate to check RO 18:04 UTC Changed BP filter for mode 26 from FM1 to FM8. Changed the set frequency to 207.8 from 207.7 Changed set frequency for mode 27 from 244 to 240. 18:08 UTC Turned on output to mode 27 damp filter 18:15 UTC Changed BP filter for mode 26 back to FM1. Changed the set frequency to 209. 18:19 UTC Daniel to LVEA Changed set freq for mode 18 from 207.9 to 207.8 18:45 UTC Daniel back 19:02 UTC Struggling to lock PRMI. Starting initial alignment. 19:10 UTC G Wave candidate alert 19:14 UTC Daniel to floor (WP 6294). Says I am going to lose the MC. Aborting initial alignment and setting ISC_LOCK node to DOWN. 19:43 UTC Daniel back. Restarting initial alignment. 20:19 UTC Initial alignment done 20:41 UTC Stopped at DC readout. Diagnosing PI and SDF. 22:44 UTC PI settings reverted. Going to NLN.
Discovered that the error point of the first ISS loop was changed to PDB about 23 hours ago. Why? Changed it back to PDA.
J. Kissel I'm not sure why, but INJ EPICs "settings" channels that are used to store, news, state and time information have become re-monitored in the SDF system, which means that the CAL-CS model shows differences when an external alert arrives and/our when a standard hardware injection goes through. I had removed the monitoring of these channels before O1 had started last year -- see LHO aLOG 21154 -- so I don't know why these have suddenly come back into monitoring. Merp. They're back to being ignored by the SDF system as they should be.
Matt Daniel Fil (WP 6294)
The MCL and MCF readbacks have a 10/100 Hz Sallen-Key whitening stage which amplifies the high frequency spectrum to get above ADC noise. Since a while we have observed a 20-50 mHz/√Hz flat noise level in the these spectra when we are locked with the IMC only. Looking with the oscilloscope we estimated about 10m V signal between 100 kHz and 1 MHz, before the whitening. This seems too much for the AA board, so we included additional low pass filters in the readbacks with cut-offs around 15 kHz. A 15/150 kHz pole-zero was added to the Sallen-Key, and another 15 kHz pole was added to the output stage.
In detail (common mode board, IMC, s/n 1102626):
The attached spectra now show a frequency noise level which is compatible with the one observed in full lock. The coherence is also improved. The ADC noise is not too far away in regions with reduced coherence.
Here is a comparison between MCF fully locked and 2W IMC only (REF traces). The changes are much smaller now, indicating that MCF sees frequency noise from the laser.
The IMC shot noise limit here should be about 1 mHz/Hz1/2, assuming 0.3 mW of light (mostly carrier) on the PD with the IMC locked, 5 mW of light on the PD with the IMC unlocked, and a modulation depth of 0.01 rad.
WP#6255 and WP#6293 completed 0930 hrs. local -> Valved-in RGA turbo to RGA volume and energized filament. 1130 hrs. local -> Took scans of the RGA volume with and without cal-gases -> isolated RGA turbo from RGA volume -> combined RGA volume with X-end volume and took scans of the X-end with and without calibration gases (inadvertently dumped ~5 x 10-4 torr*L of Krypton or 2 hrs accumulation @ 5 x 10-8 torr*L/sec into site) -> vented RGA turbo and removed from RGA hardware -> installed 1 1/2" UHV valve in its place -> Pumped volume between two 1 1/2" valves to 10-4 torr range before decoupling and de-energizing all pumps, controllers and noise sources with the exception of the RGA electronics which was left energized and with its fan running 24/7. Leaving RGA exposed to X-end, filament off and cal-gases isolated. Will post scan data as a comment to this entry within next 24 hrs..
Here are the scans from yesterday: Note the presence of amu 7 obviously "sourced" from the N2 cal-gas bottle. I will need to revisit the noted observation of the appearance of amu 7 when the cal-gas isolation valve used with Vacuum Bake Oven C is closed and the baffling disappearance of this amu when the isolation valve is opened???.
Guardian won't move on, got a message telling me to check fast shutter trigger. The "Fast Shutter Plot" button doesn't work so I manually made some plots following alog29689 (at least one DQ channel didn't exist so I picked something close). It seems to me like the shutter did not close during the last lockloss at Increase Power.
I couldn't find an alog explaining how to clear this message and make Guardian move on again.I found an instruction on how to test fast shutter on H1 Troubleshooting page. Follow the instruction by selecting LOW_ARM_POWER on LOCKLOSS_SHUTTER_CHECK guardian (had to be done manually, there's no path from SHUTTER_FAIL to LOW_ARM_POWER). Then I selected TEST_SHUTTER on FAST_SHUTTER Guardian. And got a message "Fast shutter failed tests! Do not power up!!"I think I broke the interferometer =(