Jenne, Sheila
Today we wanted to get the SRC alingment under control, mainly motivated by the observation that both our noise couplings and noise change a lot over time (within and between locks) (see 29984)
SRC ASC
We have been able to close ASC loops for the SRC at full power, both pit and yaw. We used to use AS_C to feedback to both SR2 and SRM, (SRC2), tonight we are sending this signal only to SRM, and we are using ASA45I to control SR3 angle. In the past we had a cage servo runnning for SR3 pit, but this is now disabled in full lock. I used a gain of +100 for pit and +30 for yaw, but both of these loops are verry slow as is and should probably have their gains increased a lot. (input and output matrix elemetns are all positive as well). I added some code to the guardian to use this in SRM)HIGH_POWER_ASC, but we should probably try it at 2 Watts, increase the gain and check the loop shaping, and try to power up with this configuration. I'm levaing the IFO undisturbed, so hopefully we can see if our noise is any more stable tonight.
REFL WFS detour
Since we were interested in using REFL 45 WFS as signals for SRC alignment, we wanted to prevent the 200 mHz PIT oscialltion which has dominated all REFL ASC for many months now. It turns out that even without this oscialltion, we didn't see a good signal for SR3 in these WFS, but prevetning the oscialltion seems to have helped stabilize carrier power in the interferometer.
Fixed oscilations in REFL DC centering loops
I noticed earlier that the oscialtion around 200 mHz that is present in all of our REFL WFS signals seems to come from the DCP1 loop. Jenne and I measured the loop, and found that indeed it had not much phase margin at all and a ugf around 200 mHz. The only alog we found about these loops is 9966 (from 2014). It seems that since then, someone added rather aggresive 0.9 Hz low passes that ate too much phase for all of the REFL centering loops.
We replaced the filter design for these loops with the filters used for AS centering, which allow us to get bandwidths of 2.5 Hz with about 30 degrees of phase. We also copied over the 20Hz low pass, but haven't engaged it.
The attached screenshot shows the new loop shapes, the second shows the settings used when we measured these. We do not understand why DC2 (which used REFL B DC for a combination of the 2 tip tilts) seems to fall as 1/f^4 after the resonsance.
Increased gain of PRC1 P
With the DC centering loops no longer oscillating, the refl WFS signals were slightly better but still moved alot at microseism frequencies. We found that we could improve the phase margin in the PRC1 P loop, and reduce these fluctuations, by increaseing the gain in this loop by a factor fo 4 (digital gain of 60). This was what we had set the gain to in February, (see measurement attached to 26225), but it has slowly been decreased since then. The third attached screenshot shows the refl WFS pitch signals, the PRC1 gain was changed at -10 minutes.
Lisa, Sheila
Since yesterday afternoon, we have struggled with locklosses that happen 5 to 15 minutes after we power up.
There are a dramatic, very short, glitches in DRMI LSC channels. It also shows up in the ASC channels, but after LSC.
Some lockloss times are:
2016-09-27 06:40:33 UTC
2016-09-27 05:39:35
Some times (within a few seconds) when saw the glitch but we did not loose lock are:
1158999031
1158998954
1158999031
As a test to see what the problem could be, I commented out the lines in REDUCE_MODULATION_DEPTH where the POP9 to SRCL matrix element is changed. With this matrix element out, I didn't see these glitches, but the problem did seem intermittent so this could be a coincidence.
Kiwamu, Sudarshan, Jenne, Darkhan
Overview
EPICS records that are used for calculating DARM time-dependent parameters ("kappas"), were updated using corrected DARM model (with the correct sign of the ETMY_L3_DRIVEALIGN_L2L gain). These EPICS values result in reasonable kappa values (see details).
"512 Hz DAQ downsampling" filter was installed into CAL-CS synched oscillator that replicates 35.9 Hz cal. line (ESD).
Investigations showed that the synched oscillators for 35.9 Hz cal. line were running at 180 degrees out of phase w.r.t. each other. They got synched to the same phase after I played some with the oscillator settings in CAL-CS model.
Jenne noticed that today fC was oscillating between 320 and 360 Hz at the time-scale of ~20s. This issue was resolved by turning on low-pass filters in the CAL-CS model.
Details
Sudarshan confirmed that kappas calculated from SLM tool data using these EPICS values are within reasonable ranges. After updating EPICS records one of the issues was that κTST calculated in the front-end was around -1.0. Further investigations showed that the synched oscillators for 35.9 Hz cal. line in SUS-ETMY and CAL-CS models were running at 180 degrees out of phase w.r.t. each other. We could get rid of the discrepancy by setting the phase of the CAL-CS oscillator to 180 degrees (see attached plot).
After changing settings on the synchronized oscillators their phases somehow got synchronized. So, I removed 180 degrees of an additional phase in the CAL-CS oscillator. It is still not clear what was the cause for the phase of two synched oscillators being exactly 180 degrees off. Now the oscillator outputs (after the 512 Hz DAQ downsampling) are pretty much the same (TF measurement at 35.9 Hz is attached).
New EPICS values and corresponding logs were commited to calibration SVN. The values were accepted in SDF_OVERVIEW.
Tagging CDS so they can help address the bug in the synchronized oscillators!
Title: 09/26/2016, Evening Shift 23:00 – 07:00 (16:00 - 00:00) All times in UTC (PT) State of H1: Almost back to DC readout Commissioning: 0:30 MarkP back from MY 1:30 Robert to HAM6 to plug in an accelerometer, OMC was able to lock while he was in the area
Since we seem to be needing to help the BS's alignment after locklosses, and if left alone the BS drifts in pitch for many minutes after a lockloss from high power, I've modified the filters in the M1 filter bank such that the pitch ASC values get cleared slowly, rather than instantaneously.
Rather than having a pure integrator, I've added a zero to the pole at 0Hz, and another filter with a pole at that zero's freq. This way, we can toggle the first filter module, and it'll effectively slowly clear the history of the integrator. Hopefully this will make re-acquiring PRMI / DRMI easier - operators should let me know if they think it helps or hurts relative to the last week or so.
Atleast half of the shift had H1 at Nominal Low Noise!
Day's Activities:
Locking Notes:
After the long lock this morning, H1 was stuck at CHECK_IR. Adjusted COMM OFFSET, but could only get the X-arm up to 0.5; during one attempt the XARM gain was oddly at 0(!). Opted for an Initial Alignment.
INITIAL ALIGNMENT notes:
At ALIGN_PRM, AS_AIR video looked ugly & would not lock up. Errantly moved PRM & PR2 to a time during the long lock....but this cancels the INPUT_ALIGN pointing we had (!). Kiwamu returned PR2 to it's aligned value and then continued with Alignment.
Ops Locking Note: (OpsInfo!)
I didn't get to all the alogs over the weekend. Jenne filled me in on a new Guardian state (CHECK_MICH_FRINGES) for those times when you are locking DRMI and alignment looks bad/ugly & when there is hardly any flashes. CHECK_MICH_FRINGES allows you to optimize your BS alignment for DRMI locking. Please see Jenne's alog & also the new entry in the Ops Sticky Note wiki.
(Sheila, Corey)
Violin Mode 2nd Harmonics
During a lock this morning, looked at a few rung up 2nd Harmonic Violin Modes. Below are the ones which were easily noticed to be large. The one addressed today was the one at ~1009.03Hz
1003.66 Hz: ETMX
1003.77 Hz: ETMx
1009.03 Hz: ETMy
1009.44: ETMy
Posted the 7 Day Optical Lever trends for the past week. Several Lockloss events during the week, but otherwise most trends are flat. The one possible exception is the BS which shows a slight negative trend.
Here is a summary of some of the best curves obtained in the last few days (past Thursday, Saturday, Sunday), with corresponding UTC times for reference. There are large variations in the noise from lock to lock, and also during the same lock (see for example beginning and end of last night ~7h long lock). The obvious thing that we are not controlling right now is the SRM alignment, so recommission the SRM ASC is the next step. Note: also, we haven't been consistently setting the IMC WFS offsets to minimize the peaks between 200Hz - 1kHz.
31 seconds for LN2 to be evident at the exhaust after opening the LLCV bypass valve 1/2 turn. Next, over-fill to be Wednesday
Jeff B reported lockloss at [23:41 (16:41) Lockloss – Mag 6.3 EQ near the Philippines] If it was indeed this EQ, this is certainly the surface waves producing the lockloss. At this distance the P & S waves have come and gone and the Surface waves began arriving minutes eariler based on general rules. The attached shows the corner station ground seismo exhibiting higher-amplitude lower-frequency components 10 of seconds before the lockloss.
Unlike the closer 6.2 near Japan, this one made it through the P & S wave arrivals. What a difference 1500 miles makes! Also, the moment tensor view indicates this was a Strike/Slip fault. The near-Japan EQ was a thrust fault... This could be an important component of the IFO response.
The 9MHz modulation depth reduction is now in guardian, just before NomLowNoise. It'll take the modulation depth down by 6dB.
Currently, the 45 MHz is only being reduced by 3dB.
Everything should be reset for relocking in the Down state.
Sebastien, Dave:
Last Thursday we ran seismon in continuous mode, using the script seismon_run_traveltimes. This did not work well and missed several earthquakes (the EQ data was indeed stored and running seismon_traveltimes manually detected the events correctly). We also found that the earthquake processing time per event was very long, about 90 seconds which is much longer than that observed at MIT. Investigation is continuing.
Summary of LHO DAQ over the past week. Prior to last Tuesday the DAQ had been stable for several weeks. Starting Tuesday evening (9/20) h1fw0 and h1fw1 became slightly unstable. They sometimes reported errors normally associated with QFS file system problems, and h1fw2 (non-QFS) was stable. Thursday luch time we power cycled both Solaris QFS/NFS machines (h1ldasgw0,1) which has helped with QFS issues in the past. After this point fw0 has been stable, fw1 has restarted twice in 4 days, last restart early Friday morning.
All three writers are running RG3.2 code, fw0 and fw1 have identical hardware. Our attempt to clone fw0 from fw1 last week did not work, we may want to try this again since fw0 is still asking for retransmissions and fw1 is not. (fw2 did ask for a moderate number of retransmissions last week, none since then).
model restarts logged for Sun 25/Sep/2016
2016_09_25 13:14 h1calcs
2016_09_25 13:14 h1iopoaf0
2016_09_25 13:14 h1ngn
2016_09_25 13:14 h1oaf
2016_09_25 13:14 h1odcmaster
2016_09_25 13:14 h1pemcs
2016_09_25 13:14 h1tcscs
2016_09_25 13:15 h1susprocpi
Restart of h1oaf0 models to clear DAC error which caused TCS laser trip.
model restarts logged for Sat 24/Sep/2016
no restarts reported
model restarts logged for Fri 23/Sep/2016
2016_09_23 00:24 h1fw1
unexpected restart of fw1 (following Thursday solaris restarts)
model restarts logged for Thu 22/Sep/2016
2016_09_22 03:44 h1fw1
2016_09_22 05:56 h1fw1
2016_09_22 06:08 h1fw1
2016_09_22 09:05 h1fw1
2016_09_22 10:44 h1psliss
2016_09_22 10:48 h1broadcast0
2016_09_22 10:48 h1dc0
2016_09_22 10:48 h1fw0
2016_09_22 10:48 h1fw1
2016_09_22 10:48 h1fw2
2016_09_22 10:48 h1nds0
2016_09_22 10:48 h1nds1
2016_09_22 10:48 h1tw0
2016_09_22 10:50 h1tw1
2016_09_22 12:03 h1fw1
2016_09_22 12:25 h1fw1
2016_09_22 12:28 h1fw0
early unexpected fw1 restarts. PSL ISS model change with associated DAQ restart. h1fw1/h1ldasgw1 power cycle followed by h1fw0/h1ldasgw0 power cycle. h1fw1 unexpected restart while h1fw0 was down.
JeffB reported IFO Lockloss Thursday from 6.2 EQ near Japan.
Based on the timing and the cavity build up, it looks like the vertical P wave is the strongest candidate for the lock loss. This is a pretty big EQ and only ~4400 miles distance. At this distance, the P-Wave travel time is maybe 12 minutes and the S(shear)-wave TT is about 20 minutes.
The two attached plots shows a transmitted light sum and the CS ground seismometers. This is a very nice example of classical earthquake ground motion with distinct arrivals of the P, S, and surface waves. The second attachment has the transmitted light sum overlain on the seismos. The vertical correlates well and shows a stronger signal although the X signal on the seismo also shows color at the loss time. This makes sense as the P-Wave is arriving from depth (Z) and the great circle route is very much from the NW direction(X).
This is news as we were thinking our EQ LLs were primarily from surface and shear waves. I still think that is mostly true but closer larger EQs may be P-Wave problems.
Unfortunately after swapping out the 16bit-DAC card in h1oaf0 almost two weeks ago, yesterday we got another zeroed-dac event. At 11:53 PDT, Sunday 25th September, the h1iopoaf0 model recorded an ADC, DAC and DK error in its STATE_WORD and from then onwards was correctly outputting zeros to all DAC channels. As usual, this was instructing the TCS chillers to drive to a low water temperature, and 52 minutes later the chiller tripped the TCS lasers (12:45 PDT).
Because we received both an ADC and DAC error, we could try swapping the first and second ADC cards.
T.J. has made his EPICS alarm announcement for this event more insistent, repeating every 2 minutes. If the control room can restart the models on h1oaf0 in time, we think the laser trips could be averted.
It's been a suspicion that, lately, switching DBB shutters has been causing IMC to lose lock. Perhaps this action is causing just enough noise to perturb the injection locking? Until further investigation is done, scans will be taken at times when the mode cleaner can be down for a period of time such as maintenance days.
Weekly Xtal - No changes from last week
Weekly LASER - No changes from last week
Weekly Env - RH for ANTERM, LASERRM and DIODERM are higher than last week. All temps seem to show no change.
Weekly Chiller - nothing unusual except for the change made to the flow rate on 2/19
Given the number of trips over the last week (which are easily seen on the above attached trends), everything looks normal.
Trouble with MODE17 (ITMX 15542 Hz) could be due to instability from large phase changes with slight frequency drift in our damping loop. Mechanical mode frequency at beginning of a cold lock sits around 15541.875 Hz and has drifted to 15541.95 Hz after about three hours. The previous bandpass filter carried ~40 deg phase change over that time. In an effort to fix this, I've made two BP filters that are shifted versions of one another to be turned on as the frequency migrates. The guardian uses the PLL frequency monitor to track the average frequency change of the mode peak and make the filter change switch at the appropriate time. Fall offs and notches in the filters are such that center of BP --> switching point ~10 deg. I've watched the guardian successfuly make this change.
Proof of concept appears to have worked; Mode17 was kept stable throughout frequency drift well beyond the previous 3 hourish mark. Lockloss occured after ~6 hours from the partner Mode25 who saw a phase change of ~140 deg over this time, eventually probably driving up this usually easily dampable mode. Attachment shows freq drift over last night's lock. I've added stepping bandpasses to both modes (controlled with the guardian) now to accomodate frequency drift over much longer locks.