TITLE: 11/03 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Broken?
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Earlier during the night we had issue with instability during Engage_SRC_Part2 (addressed by Sheila -- see alog31158). Later the IFO kept losing lock at Increase_Power. Since lockloss tool doesn't work I couldn't pull up the log prior to locklosses. Visually from the control signals and error signals nothing was going unstable. The lock just dropped as the power increased about the same place everytime. After the third lockloss Guardian complained about fast shutter. The test failed and I couldn't move on with the locking sequence. Richard went out to the rack and mentioned that he didn't get a good voltage reading when the shutter was opened but got okay reading when the shutter was closed. We checked GS13 to make sure that it saw the shutter motion and it did. However, the SYS channels that supposed to report shutter state and voltage didn't see anything (maybe this is why the shutter test failed?). So to my understanding the shutter is somewhat functioning. Anyway, right now it seems like the ISC_LOCK guardian is stall because of the FAST_SHUTTER guardian. I was able to make it all the way to DRMI_ASC_OFFLOAD before it refuses to move on.
Ops Info: On an unrelated note, violin modes have been coming back high every time after locklosses. At one point it saturated OMC DCPD. So be mindful before moving on pass DC_READOUT_TRANSITION.
There were no sign of anything going unstable. It's just dropped. I tried engaing the new PRC1 filter that Sheila removed from the Guardian as IFO goes to high power but it's still causing instability at that point.
Guardian won't move on, got a message telling me to check fast shutter trigger. The "Fast Shutter Plot" button doesn't work so I manually made some plots following alog29689 (at least one DQ channel didn't exist so I picked something close). It seems to me like the shutter did not close during the last lockloss at Increase Power. I couldn't find an alog explaining how to clear this message and make Guardian move on again. I found an instruction on how to test fast shutter on H1 Troubleshooting page. Follow the instruction by selecting LOW_ARM_POWER on LOCKLOSS_SHUTTER_CHECK guardian (had to be done manually, there's no path from SHUTTER_FAIL to LOW_ARM_POWER). Then I selected TEST_SHUTTER on FAST_SHUTTER Guardian. And got a message "Fast shutter failed tests! Do not power up!!"
I think I broke the interferometer =(
Tonight (Nov 3rd around 8:20 UTC) we saw something none of us had seen before, ETMY violins rang up very suddenly. The noise is DARM was extremely high, and we saw beat notes between violins at low frequencies in DARM once things calmed down.
Nutsinee has been damping them very effectively, but we don't know why they rang up so sudddenly
I have made scripts that we can run that will adjust the a2l gains for PR2 and PR3. These use the same a2l engine that the test mass script does, it just sets the settings for PR2 and PR3. They can be found in the same location (just checked in): .../userapps/isc/common/scripts/decoup/a2l_min_PR*.py, where * is either 2 or 3.
Each script has been run tonight, so we now have non-zero a2l elements for both those mirrors.
Nutsinne is having trouble relocking, with instabilities around 3.5 Hz that ring up when she engages PRC1 ASC. We removed both the new offset from POP A pit and the new a2l from PR3, hoping that this will help.
Stefan's offset for POP A was -0.05, Jenne's a2l coefficients were PR3_M3_DRIVEALIGN P2L 0.85 and Y2L 1.85
The problem was the new cut offs added to the PRC1 loop tonight, they seemed fine at low noise but aren't stable when the loops forst come on. We have just removed them from the gaurdain for now.
Hmmm, odd. Anyhow, the cutoffs now come on in LowNoiseASC rather than earlier.
I tried to set up an acoustic injection that reproduced several of the bumps in DARM (last time I only did one shaker bump) and tried PR2 pitch, yaw and length injections and IM2 pitch. The best was PR2 pitch, see attached figure. The behavior at other bumps was similar.
x5 scatter reduction at 50Hz.
=======================
Taking Robert's observation as a hit, we moved the PR2 spot position in PIT (using the PR2_SPOT_MOVE state), looking for variations in scatter coupling while driving with acoustic noise.
Indeed, as we move from -275urad to -302urad PR3 PIT alignment counts, the scatter peak at 50Hz varies like a bar code reader.
The starting alignment (PR3 PIT = -296.72urad) corresponds to a maximum in scatter at 50Hz . We found minima at -301.72, -299.32, -292.47, -290.22 and -278.52 urad or PR3.
We thus picked -292.47, which seemed to be the best perfoming spot. It corresponds to a PR2 PIT alignment of 1767.24urad (see alignment snap shot).
The attached plot 1 shows the undriven DARM noise (blue), the acoustically driven DARM noise at the original position (-296.72urad of PR3 PIT), and the acoustically driven DARM noise at the final position (-292.47urad).
Plot 2 is an alignment snap shot.
Finally, to lock this position in, we had to add a POP_A_PIT_OFFSET of -0.05. Jenne ran a PR2 A2L script (demodulating PRCL) to mark the new PR2 spot position. The required M3 drive align numbers are in snapshot 3.
I ran the current version of the calibration pipeline over a stretch of O1 data to reproduce the kappas and compare to those in the C02 frames. The filters file used was aligocalibration/trunk/Runs/O1/GDSFilters/H1DCS_1131419668.npz, as suggested by the calibration configuration page for O1: https://wiki.ligo.org/viewauth/Calibration/GDSCalibrationConfigurationsO1#LHO_AN2 The agreement looks quite good. Time series plots of the kappas and the cavity pole are attached. The start time used here was 2016-10-04 12:41:19 UTC (GPS 1127997696).
alog 30897 has links to SRCL, PRCL and MICH BruCo's. Here hare some interesting highlights from those graphs:
SRCL:
Sure looks like our SRCL sensing is limited by input YAW jitter below maybe 25Hz.
BTW: MICH sees the same noise: CAL-CS_MICH_DQ
PRCL:
This looks like all our PRCL noise is jitter in the recycling cavity...
( ASC-POP_A_NSUM_OUT_DQ ... PIT is more than SUM ...
ASC-POP_A_YAW_OUT_DQ ... or YAW. )
Also interesting:
ISI-HAM3_SUSPOINT_MC2_EUL_L_DQ
I attached all plots for archiving purposes.
Out of curiosity, I looked at coherences of channels listed above from 24Oct2016 (Stefan's data above), and 24Jan2016 (a time when Gabriele ran a BruCo, so was probably a good lock time).
In the attached screenshot, the right half is coherences, and the left half is spectra of those channels. On the coherences half, top right is SRCL vs. IMC-MC2_YAW (ASC control signal), bottom right is SRCL vs. WFS_B_Pit (ASC error signal), top left is PRCL vs. POP_PIT and bottom left is PRCL vs. POP_SUM. On the spectra half of the screenshot, the top row is the length DOF signals, and the lower 2/3 of the plots are in the same locations as the coherences plots. In all plots, green is now-ish and blue is O1 config.
[MattE, Jenne]
We put in modification filters to the IMC WFS loops to move the existing lowpass filters from ~8Hz down to ~1Hz. Both filters (FM9 and FM10 in each DoF) must be on to actually get the full lowpassing.
This didn't change the SRCL noise, but it did reduce the coherenece with the MC2 angular control signal. In the attached screenshot, red is from earlier today (note that it is much less coherent than the green traces from 24Oct2016), and brown is after turning on these new lower lowpasses. I removed the blue and green traces from earlier to avoid too many traces on the plot, but the brown now is basically just as good as the blue from January.
Ed, Patrick and Jenne
Apologies if this is a repeat, but I've never seen this and was at a noticeable loss for action to take.
After the Lockloss at 23:30UTC, there was a diag main message that indicatd that ISC_LOCK wasn't going anywhere until the Fast Shuttter was tested. I'm making a Wiki in Troubleshooting H1 to address what to do in this situation. Jenne came to the rescue. Thanks Jenne!
Took well over 60 minutes to overfill CP3 by doubling LLCV to 36% open first, then open a little bit more, then a bit more, until finally it overfilled, I will change the setting LLCV from 18 to 20%.
Matt E., Dave, Jenne, Jeff K. and I tracked the problems with PI down to settings not being restored or not being restored correctly by SDF after the reboot on Tuesday. It is unclear if the problem with SDF was human or computer error. Jeff K. and Matt reverted the PI settings to those from a week ago. It appears that mode 27 is now back under control. 15:00 UTC Robert to end X to perform injections while at NLN 15:52 UTC Lock loss (PI mode 27). Tried changing sign of gain and turning off damping to no effect. 15:52 UTC Christine and Karen to mid stations 15:54 UTC Kyle to end X 15:54 UTC Robert back 15:56 UTC Robert to end Y to pickup hardware 16:15 UTC Changed to VERY_WINDY_NO_BRSXY ? Robert back 16:19 UTC Pausing at violin mode damping 2 waiting for roll modes to damp. 16:33 UTC Kyle back. Checked that RGA does not have a leaky valve. It does not. 16:50 UTC Karen leaving mid Y 16:52 UTC mode 27 starting to rise 16:59 UTC Lockloss Changed to WINDY during CHECK_IR 17:46 UTC Stopping at VIOLIN MODE DAMPING 2 again. 17:59 UTC King soft water through gate to check RO 18:04 UTC Changed BP filter for mode 26 from FM1 to FM8. Changed the set frequency to 207.8 from 207.7 Changed set frequency for mode 27 from 244 to 240. 18:08 UTC Turned on output to mode 27 damp filter 18:15 UTC Changed BP filter for mode 26 back to FM1. Changed the set frequency to 209. 18:19 UTC Daniel to LVEA Changed set freq for mode 18 from 207.9 to 207.8 18:45 UTC Daniel back 19:02 UTC Struggling to lock PRMI. Starting initial alignment. 19:10 UTC G Wave candidate alert 19:14 UTC Daniel to floor (WP 6294). Says I am going to lose the MC. Aborting initial alignment and setting ISC_LOCK node to DOWN. 19:43 UTC Daniel back. Restarting initial alignment. 20:19 UTC Initial alignment done 20:41 UTC Stopped at DC readout. Diagnosing PI and SDF. 22:44 UTC PI settings reverted. Going to NLN.
Discovered that the error point of the first ISS loop was changed to PDB about 23 hours ago. Why? Changed it back to PDA.
(Carlos, Richard, Gerardo)
Y-End RGA is ON.
By "on" I assume that the electronics are energized (i.e. the fan is running) but not the filament?
That is correct, Kyle.