The laser tripped this morning. The status screen red flagged the following:
- Xtak chiller flow
- Interlock OK
- Frontend Power error (WD)
- Oscillator Power error (WD)
Initial thought was that a water leak occurred. That was not the case.
Suspect that the power watchdog was tripped because the injection locking was lost.
When the injection locking servo loses lock, the laser power drops and sets off the
watchdog.
The laser restarted without any problems, other than the alignment into the pre-modecleaner
was off a little. Upon recovery, the pre-modecleaner had some problems locking. Went into the
PSL Enclosure to adjust the alignment but by the time we got inside, the pre-modecleaner was locked.
Tweaked the alignment a little.
Closed the other loops successfully. After deliberately bringing up and down the input
modecleaner a couple of times, the FSS seemed fine with the occasional PZT oscillation.
JeffB/Ed/Peter
Sheila, Matt, Kiwamu, Carl, Stefan
Earlier today we tried heating TCS CO2 X-arm with 2Watt (0 Watt into Y), and all we saw was a futher drop in recycling gain.
Tonight (07:29 UTC) we tried the opposite TCS: TCS CO2 Y-arm with 1Watt (0 Watt into X). (Half of what we put into x, because we broke lock on the first try.)
Result: Absolutely nothing - all recycling gains remained the same or further dropped.
Conclusion: TCS CO2 cannot get any recycling gain back.
============================================
Log:
UTC 20160713 23:08:09 all TCS CO2 completely off
no effect on any sidebands
UTC 20160713 23:16:22 TCS CO2 X to 1 W
UTC 20160713 23:21:21 TCS CO2 X to 2 W
recycling gains drop, lock loss
UTC 20160714 07:29:00 TCS CO2 X to 2 W, TCS CO2 Y to 0 W
recycling gains drop, lock loss
This lock was brocken by ITMY bounce mode. It looks like we may need to double check all the bounce mode phases to see that they are unchanged at 40 Watts.
There were several problems with PI damping this evening, the recovery of the safe.snap after ETMY's failures today resulted in H1:SUS-ETMY_BIO_L3_PI_ULLL_SW selecting the UL quadrant rather than the LL quadrant resulting in no ETMY damping for the entire evening.
In the 50W lock an ITMY mode (purple trace -80min) reached saturation level, the resulting unstable bounce mode at lock loss may have been associated. The mode was still at elevated amplitudes at the beginning of the next lock but was damped. The damping phase needs careful monitorring a changing sign regularly. I suggest moving to QPD error signals. Finally at -15min I set H1:SUS-ETMY_BIO_L3_PI_ULLL_SW to LL and was able to damp ETMY modes again.
For the bounce mode phasing: is the DARM gain now high enough that the cloop is close to +1 at the bounce mode frequencies?
I've put together some plots of the common and differential lenses as measured by the Hartmann Sensors.
The only change I made to the Hartmann sensor data is to scale the ITMY spherical power (H1:TCS-ITMY_HWS_PROBE_SPHERICAL_POWER) by (17.5/7.5)^2. This is because the magnification has defaulted to 17.5x on HWSY instead of 7.5x which is the actual level. Additionally, I reset both HWS measurements to zero at t=0.
The first plot (LHO_diff_CO2.pdf) shows the HWSY and HWSX measurements as well as common ([X+Y]/2) and differential (Y-X) lenses along with corresponding CO2 laser powers.
The second plot (gain_vs_time.pdf) shows an approximation of the gain: the TR-X_SUM / IMC_OUTPUT. Perhaps someone can point me to a better approximation. You can see the reduction in gain powering up. The very sharp initial spikes can be ignored.
The third and fourth plots show the gain vs common and differential lenses, respectively. Remember, there is an offset in lensing in both of these plots due to resetting the HWS values to zero.
It's certainly not conclusive, but:
Quick Summary: Commissioning. PR gain and noise hunting. Carl is working on PI. We stayed locked at 40W for a while. Shortly locked at 50W before Bounce mode and PI broke it.
Log (All time in UTC):
0:04 ETMY ISI (PAYLOAD WD) tripped. ETMY SUS is frozen. Dave restarting SUSEY IOP.
4:33 Sheila to ISCT1 aligning POP diode
4:49 Sheila back
~05:05 Ran A2L
- Something interesting happened: As PI started ringing up (ITMY 14979Hz and ITMX 15521 Hz) , moving SRM in the corret direction in order to improve AS90/POP90 helped keep PI under control.
Nutsinee, Sheila
There is something wrong with the REFL and POP cameras, we see no image from them although we shoudl be seeing something on REFL.
Actuallythe POP camera is just misaligned (probably has been since we inserted the beamsplitter monday), but there is still nothing on refl which I think there should be.
The issue with the POP camera was found to be a clipping at the newly installed beamsplitter. This was fixed today. The REFL camera was completely misaligned for some reason, perhaps it was mistakenly moved around. The REFL camera was also realigned. So both are back to functional.
Stefan, Matt, Kiwamu,
The online calibration, aka CAL-CS, is more accurate now; the sign of the simulated ETMY L3 stage (CAL-CSDARM_ANALOG_ETMY) was found to be wrong and we fixed it. Fixing the error resulted in an improved noise level at around 100 Hz in CAL CS. This should not affect the GDS pipe line calibration. The attached shows a comparison of CAL-CS before the fix and an offline calibration using DARM IN1 and the full DARM model (28179). It is clear something wrong was going on in 40 - 200 Hz.
What we changed:
This fix gave us an actuator model which is consistent with the measurement (28179) in the sense that the relative phase between the PUM and TST have a relative phase of 180 deg at high frequencies. Also, traditionally, when ETMY had a positive bias, the gain of the L3 stage used to be set to -1 in the O1 era (see for example 25575). Therefore today's fix is consistent with the O1 era too. One thing I still don't understand is the relative calibration difference between GDS and CAL-CS (summary page). The relative magnitude should show of a factor of 2 difference or so around 100 Hz assuming the sign error was only in CAL-CS, but it does not show such a big difference. Not sure why.
The online GDS (called C00 during O1) calculation uses CAL_DELTAL_CTRL and CAL_DELTAL_RESIDUAL to produce h(t). Compared to the front-end, It applies better relative timing between the two signals and other high freqency corrections. Since CAL_DELTAL_CTRL is obtained after the application ANALOG_ETMY_L3 filter, the online GDS will also have the same problem as front-end DARM signal. Only the offline GDS (called C01, C02 during O1), uses CAL-DARM_ERR and CAL-DARM_CTRL along with actuation and sensing models to produce h(t) and hence would have been different. I am not sure whether we have produced that at this point.
Thanks, Shivaraj.
You are right. I misinterpreted the subway diagram (G1501518-v10) last night
. I agree that C00 must have the same sign error and therefore what we saw in the summary page is correct.
The scipt, which produced the comparison plot, is saved into svn, so that one can use the code in some future when it is needed. The code lives at
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER9/H1/Scripts/ControlRoomCalib/H1CalibDoubleCheck.m
Kiwamu, Matt, Stefan
We were still suspicious of the 2-omega RF buildup signals, so we decided to put a line on the PRM length at 303Hz, and demodulate the line. To avoid loop feed-back we turned on the existing "notches" FM8 filter in PRCL1. POP9_I was demodulated at 303Hz in H1:LSC-LOCKIN_1_DEMOD_3_I_OUTPUT.
Assuming good overlap, we would expect the gain to scale as sqrt(carrier recycling gain * 9MHz recycling gain).
Interestingly, the 9MHz sideband drops by about the same amount as the carrier, but with a timne constant of about 10min (vs about 1 min for the carrier).
The attached plot shows the power increase to 40W, without the soft offset adjustment that usually happens right away. All scales (except purple) start at zero at the bottom of the plot.
Conclusions:
- POP18 is not reliable - it suggest a larger gain drop than the optical gain.
- The optical gain (RF9 & carrier) does drop by the same 25% as the carrier, but on a much longer time constant.
Following my talk at the Systems meeting, I've added some pdfs to the DCC entry with 'publicity plots' of the improvements due to the BRS: G1601529: Tilt- and Sensor-correction filter tuning at LHO. The figure of merit I apply to these plots is that we want to reduce RMS velocity. I believe this is the correct figure of merit from DC up until at least the lowest Quad resonances. This saves us the worry of trying to compare bumps and dips in spectra. In all the following plots, the dashed curves represent the high-to-low RMS of the solid curves of the same colour. Most of them start from some cut-off frequency, to isolate the low frequency contributions. In general, the output of all inertial sensors can be regarded as noise below 0.1Hz, since it's either uncorrelated or it's something that will be common to the ends and the corner. The only new figure here 'Sensor correction vs not' suggests that the noise injection from sensor correction, about 8e-8m/s from 0.1Hz to DC, is lower than the potential benefit of sensor correction, about 9e-8m/s from 0.3Hz to 0.1Hz, even with high winds (25mph) and very low microseismic motion. With higher microseism and lower wind, this ratio will only improve. Still, during very quiet times, we may be better off disabling low-f sensor correction everywhere.
at 16:58 PDT all user models on h1susey stopped running. It looks like the IOP model developed a DAC error, and stopped running all the DAC cards (and also powered down the SUS-AI chassis). The IOP model continued to run (user models had stopped with ADC TIMEOUT errors in dmsg), so we could determine the SWWD RMS readbacks were well below trip levels. Hugh got seismic driving again by overriding the broken SUS IPC.
At 17:09 PDT I restarted all the models on h1susey (iop, etmy, tmsy and etmypi).
Title: 07/13/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT) State of H1: IFO unlocked. Peter working on FSS instability. Commissioning: Outgoing Operator: None Activity Log: All Times in UTC (PT) 14:35 (07:35) Peter – In the LVEA working on the FSS 14:56 (07:56) Peter – Out of the LVEA 15:00 (08:00) Start of shift 17:10 (10:10) Carlos – Rebooting Digital Video Server-1 17:30 (10:30) Peter – Alignment work on PMC (WP #5994) 18:45 (11:45) Hertz Equipment Rental on site to deliver snorkel lift to End-Y (WP #5995) 20:39 (13:39) Robert – Going to End-Y to setup magnetometer 21:10 (14:10) Kyle – Going to Mid-Y to overfill CP3 21:14 (14:14) Robert – Back from End-Y 22:18 (15:18) Kyle – Back from Mid-Y 23:00 (16:00) Hand off to Nutsinee End of Shift Summary: Title: 07/13/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT) Support: Peter, Cheryl, Stefan, Sheila, Keita, Kiwamu Incoming Operator: Nutsinee Shift Detail Summary: Peter working on FSS instability. Working on relocking after FSS was stable. Locked at INCREASE_POWER, 39.5W. Hand off to commissioners.
1440 -1510 hrs. local -> To and from Y-mid Opened exhaust check valve bypass valve, opened LLCV bypass valve 1/2 turn -> LN2 @ exhaust in 60 seconds -> Restored valves to as found configuration. Next CP3 overfill to be Friday, July 15th. Also, demonstrated MTP Safety Valve functionality when backed by local scroll pump (rediscovered that the "NORMOAL" light LED on the turbo controller is burned out) -> Let MTP brake phase complete but am leaving turbo rotor levitated, i.e. controller energized, overnight -> will de-energize tomorrow.
~0900 hrs. local -> De-energized Y-mid Turbo controller (Foreline 1.2 x 10-1 torr, turbo inlet 2.6 x 10-1 torr seams odd - gauges reversed?)
(C. Gray, H. Radkins, M. Robinson)
Ran through the new SEI FAMIS task, H1 ISI CPS Sensor Noise Spectra Check (this one is #6854). Will be making slight edits to the procedure, but it should be ready to add to the line-up of FAMIS tasks for LHO Operators.
This is a weekly measurement and I picked a quiet measurement time from about 2am PST. Running the measurement for the BSC & HAM ISIs did NOT show any blaring issues with the CPS. Hugh's eye was caught by a bump for the ETMx's Stage2 at around 67Hz. (Screenshot of BSC & HAM CPS spectra is attached).
Attached are pressure trends from cold cathode gauges from Jan. 2016 to now. The noise level of the gauge seems to be higher since Beckhoff system was installed.
The FSS had a hard time acquiring lock and staying locked this morning. I had noticed that the pre-modecleaner
reflected spot looked "mis-aligned". The pre-modecleaner alignment was tweaked up, bringing back the vaguely
familiar reflected spot. The reference cavity alignment was also tweaked up but really there was no substantial
improvement in its transmission.
Whilst the frequency servo was trying to acquire, I noticed that the injection locking relock counter had
jumped from ~3 this morning to over 40. In addition the high power oscillator PZT showed a few discrete level
jumps.
Even with the input modecleaner disabled, the laser frequency was being yanked as can be seen from the reflected
spot showing more than the usual two lobes. I tried re-acquiring the FSS lock at a different NPRO crystal temperature.
Previously the slow voltage slider was around -0.0010. Now it is around -0.2630. I have also changed the locking
search ramp to reflect this. Things seem to be more stable at this operating point - the input modecleaner acquired
lock for a start.
If the FSS lock is reacquired at a slow voltage close to zero and the FSS has a hard time maintaining lock, and
things seem better at a different slow voltage, this could be an age-related symptom of the NPRO. Something to watch
out for.
One could ask why the NPRO PZT and not the high power oscillator PZT. The high power oscillator PZT time series does not fluctuate wildly compared to the NPRO PZT, which is why I suspect the NPRO.
J. Kissel, S. Dwyer, S. Ballmer We continue to have trouble with the FSS oscillating after a lock loss, in that it'll often either take several minutes to relax, or it requires manual intervention such as briefly reducing the common gain of the FSS loop. As such, Sheila took a look at the IMC PDH loop to look for problems and instabilities there. I looked over her shoulder at her results, and saw some areas for improvement in the loop design. The current loop design has an UGF at 66 [kHz], with a phase margin of 68 [deg]. However the gain margin around ~200 [kHz] is pretty dismal because of what looks to be some icky features in the physical plant. These features have been shown to be directly influenced by the FSS common gain (see second attachment in LHO aLOG 28183). I figure, given that we've got oodles of phase margin, what harm could be done by just adding a simple 200 [kHz] pole in loop, and reducing the gain by ~2 [dB]? As such I took Sheila's data, which lives here /opt/rtcds/userapps/release/isc/common/scripts/netgpib/netgpibdata/TFAG4395A_12-07-2016_163422.txt (also attached) and added these modifications offline as a design study. In the attached plots, I compare the as-measured IMC PDH Open Loop Gain, G, Loop Suppression, (1/1+G), and the Closed Loop Gain, (G/1+G), against one modified as described above (blue is as measured, and green is the modified design study). The results are encouraging: a still-substantial UGF of 47 [kHz], and a very-healthy phase margin of 58 [deg]. However, as can bee seen in the loop suppression and the closed loop gain, there is far less gain-peaking and/or a much great gain margin and we would no longer have to worry about the icky features in the plant that are so sensitive to the FSS common gain. Where to stick such an analog filter? It's of course dubious to claim that the MEDM screen for such a system is representative of the analog electronics, but assuming it is, one can see that there is the possibility of a switchable daughter board in the FAST path that gets shipped off to the PSL AOM for the FSS. Because it's switchable, we can toss whatever simple filter in there that we like, and then compare and contrast the performance for ~1 week to see if it improves the stability problems we've been having. What impact would this have on the full IFO's CARM loop? I'll remind you of Evan's loop analysis of the whole frequency stabilization spaghetti monster in LHO aLOG 22188. There he suggests that the CARM UGF is around 17 kHz, so as long as the Closed Loop Gain around there is the same, then this change in the IMC PDH loop should have little impact [[I just made this sentence up based on just a few words from Sheila who asked me to look at the CLG. I'm not confident of its truth. Experts should chime in here]]. Indeed the third .pdf attachment shows that G/(1+G) of the IMC PDH loop, regardless of modification remains unity out to 100 [kHz].
How does this compare with the Pomona box from anno domini?
Description of the notch in pamona box 5141 (this was in the loop for a few years, but was removed serveral months ago, I think before O1)
Thanks for finding the aLOG entry Shiela! @Daniel -- though she doesn't say it explicitly, the aLOG shows that the Pomona Box notch was centered about ~700 kHz. As shown by my OLGTF model, if we add this ~200 kHz pole, then not only will any features at 200 kHz be suppress significantly, but whatever might happen at 700 kHz is even further suppressed. In otherwords this pole just shapes the high-frequency, super-UGF portion of the OLG to better handle *any* non-sense, instead of the focused bandaid fixes that any notch would provide.
Before the alignment tweak the output of the locking photodiode was ... - unlocked -1.017 to -1.024 V - locked -0.148 -0.161 mV This corresponds to a visibility of ~85%. After the alignment tweak ... - unlocked -1.017 to -1.024 V - locked -0.142 mV to -0.155 mV This corresponds to a visibility of ~86%.