Began 15:22 UTC, finished 15:24 UTC
As soon as the restart finished IPC was red. I hit diag reset button and it went away. OAF_L0_MADC1_EPICS_CH25 started to overflow pretty much as soon as the model restart was done.
TCS chillers and lasers are recovered.
The laser tripped this morning. From the laser MEDM screen it looked like the NPRO had tripped which would pull
the entire system down. However the Beckhoff screen indicates otherwise; that the fault is associated with the
crystal chiller. It should be noted that the crystal chiller has 10468 hours on it.
Logs indicate that it tripped around 3 am. Unfortunately I did catch the trip.
Further inspection shows that the laser tripped not because of the crystal chiller (see AMPFlow.png).
Might be due to flushing of the newly deployed (old) manifold.
Looks like the power meter circuit caused the trip this time around. Doesn't appear to be the crystal chiller, however I am uncertain how often the chiller flow is updated via the RS-232 link.
The PSL trips over the last couple weeks have all been due to the Front End flow glitching. Suspecting the FE flow sensor might be faulty we swapped the PSL water manifold for the spare, with new flow sensors installed on the spare. As Peter says above this latest trip was due to the Power Meter water circuit (first time we've had an issue with this water circuit since the end of July). Unclear at this time what the cause was, although it's likely to be junk stirred up by our manifold swap on Tuesday working its way through the system (as Peter mentions above); Peter has some pictures of the output water filter for the PSL cooling system and there are a few new occupants taking up residence.
Filed FRS #6169.
Sheila, Terra, Jamie
Today I set up an OSA on ISCT6 to look at sideband assymentry at the AS port. Mark stayed late to get an MHV cable together, and we were able to look at the asymmetry in some of the 50 Watt locks that Jenne and Stefan had. Roughly, we saw that the asymmetry of the 45 MHz sidebands is 66% in power, while the 9 MHz sidebands are about a factor of 3 different in power. We have left the set up in place for now, and we will make some plots tomorrow.
Jenne, Sheila, Matt, Stefan, Terra, Jamie, Robert, Kiwamu commissioning Still needing to stop at LOCK_DRMI_1F to minimize ASC error signals before engaging control loops. The TCS chillers tripped from a DAC error and were reset. 23:18 UTC Sheila to ISCT6 00:19 UTC Sheila back 00:37 UTC Sheila to ISCT6 02:00 UTC Sheila back 03:23 UTC Restarted frozen video2
Sheila, Jenne, Terra, Patrick It appears that we have the same situation as described in alog 27435.
We ran through the instructions in the referenced alog and brought the TCS chillers and lasers back.
The OAFIOP Dac has gone bad again. We're just going to leave it at this point. The TCS lasers will need to be turned on in the morning, after this has been addressed.
Following the suggestion of alog 29522 we switched the ASC input matrix for MICH_P and MICH_Y (the BS).
The final matrix was
MICH_P = 1.5*AS_A_36_Q_PIT + 0*AS_B_36_Q_PIT
MICH_Y = 1.5*AS_A_36_Q_YAW + 0*AS_B_36_Q_YAW
the old matrix was
MICH_P = 0.666*AS_A_36_Q_PIT + 1*AS_B_36_Q_PIT
MICH_Y = 0.666*AS_A_36_Q_YAW + 1*AS_B_36_Q_YAW
We did the ransition slowly over about 30min. Interestingly the SRM dither loop did a nice job, and the different offset in the error signal was not an issue.
We stayed at 50W a total of 40min.
While working on this input matrix switching, we saw at least locklosses that are a result of LSC_POP_A_RF45 having RF saturations. The 3rd attachment below was from a lock when Stefan was changing the input matrix a little more quickly, and we lost it during this process. The first two attachments are from our 40min lock, with different time scales.
We see that several of the RFMONs get a little high as we get close to breaking the lock, but the POP_A_RF45 is much higher than the steady-state value should be, and goes high almost a second before we lose the lock.
I didn't save any screenshots, but other locks from earlier today did not lose it due to these RF saturations, so this is definitely something that's coming up with the BS input matrix changing.
This is yet another reason that we need to revisit lowering the 45MHz modulation depth, as soon as we have a semi-stable interferometer.
Keita Marc Fil Daniel
We realized that we can implement the required compensation for the second loop by using the whitened monitor signals (TP10) from the transimpedance board as the inputs to the sum of PD1-4 and PD5-8, respectively—instead of the unwhitened outputs from the transimpedance amplifiers.
The required modifications are:
During testing we noticed that the input transimpedance amplifiers (eight AD797s), were all oscillating between 10-15 MHz with an amplitude of about 500 mVpp. Adding capacitance to the feedback network seemed to have little effect, so we swapped all of them with TLE2027. This solved the oscillation. Using 400 Ohms transimpedance, the input referred noise of a channel is about 25 pA/√Hz between 10 Hz and 10 kHz. This is maybe a factor of 2 below shot noise at high laser input power. The electronics noise is dominated by the Johnson noise of the first 4.87K resistor in the whitening stages.
The output SMA connector which was shortening the negative leg to the chassis has been removed. Instead, we drilled a new hole for an isolated TNC connector.
With the upfront whitening gain the fast monitor points now have too much gain and are saturating. We removed the gain of 50 from ERR1 and ERR2 by replacing R60 with a 4.53K (from 220K). This also removed the 2.7 kHz pole in this path. The OUTPUT channel was also modified for a flat transfer function with a fixed gain of 1. It now looks like the other two. In detail, C52 and C53 were shorted out, and R60 and R61 were changed to 4.53K (from 45.3K).
The transfer function of the ISS outer loop AC coupling is attached. As implemented it should be unconditionally stable with a ugf of 10 Hz. With a gain of ~500 at 10 Hz in the overall outer loop servo, the AC coupling point would be around 0.1 Hz.
Here is a spectrum of the outer loop readbacks at 2W and 50W input power, respectively, with the ISS second loop open. The AC coupling is on. At 50W the third loop is also on. The ERR readbacks are very near saturation at the higher power. Since the error signal is followed by a fixed gain of 3, the output was saturating.
I stopped and reran the Hartman code for ITMX at around 21:30 local.
As suggested by Aidan, I restarted the Hartman code with a fresh template. This time, I restarted both the ITMX and ITMY codes at around 9:40 local.
Note that the last restart before yesterday was done on Aug.8 (28947). It was done for both ITMX and ITMY. They have been running without errors until I stopped one of them (namely ITMX) yesterday.
We had an interferometer that was nicely thermalized at 20W today, including a 900Hz frequency line to monitor the REFL9I and REFL45I gain. So we decided to investigate the influence of common TCS on both AS_90 and REFL45 gain.
We thus increased the TCS CO2 central heating by 1W on each optic (to 1.2W on X, 1W on Y). This resulted in a DECREASE (at least initially) of the REFL45 gain (the opposite of what we see going to high power), and at the same time a recovery of the AS_90 buildup.
After letting it settle for about 50min in this configuration, we went to 50W. To our surprise the sign of REFL45I changed. We can only explain that by the 45 sideband going through critical coupling. This would mean a number of our loops in REFL changes sign, so it is not surprising that we lost lock soon afterwards.
The attached plot illustrates that story. Blue: REFL45I gain, Green: REFL9 gain, Orange: AS_90
At the same time, we monitored the intensity coupling as another figure of merit. See the attached. The time t in the legend box is initialized at the start of the test so that it starts with t = 0.
The coupling above 100 Hz initially improved by a factor of few, but then later it came back up and eventually became higher than the initial coupling.
If we compare today's data against what Evan measured back in this February (25476), the latest coupling is higher in most of the frequency band. The current coupling is higher by a factor of 10 at 100 Hz and too high by a few factor at 1 kHz. The second loop was open during the test.
PSL/IOO jitter coupling is high, so, as a first step, we adjusted offsets on IMC_DOF_1(2)_P(Y). We adjusted offsets to minimize peaks in DARM from a 290Hz pitch (1500 counts), and a 310 Hz yaw (390 counts) injection, made using the piezo mirror on the PSL table. The second loop of the ISS was off, and we were at 20 W. This was a first rough alignment (the offsets were at 0). Only DOF_2_P and DOF_1_Y seemed to have much effect. The figure shows that we improved jitter coupling by a factor of 3 or 4 in both pitch and yaw, with the best improvement in pitch. Next we will adjust the ISS array picomotors.
The proposed settings change follows:
IMC_DOF_1_P, was: 0, proposed: 0
IMC_DOF_2_P, was: 0, proposed: -100
IMC_DOF_1_Y, was: 0, proposed: 50
IMC_DOF_2_Y, was: 0, proposed: 0
Robert, Kiwamu
This is a follow-up on the issues seen by h1fw0 as detailed in https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=29452 Yesterday we installed a new build of daqd on h1fw0 and h1fw1. This daqd wrote writes more debug information to logs, to help us see diagnostics of the input stream. Looking at a frame that showed inconsistencies we saw what appears to be a series of 1/16s time slices that were flipped. After reviewing some code with Jim Warner I noticed a bug in the newer code that could allow a time slice to be reordered when there was a stall in one of the producer threads (the producer threads injest the data). This was fixed, and h1fw0 & h1fw1 were restarted. This fix will clear up at least some of the issues we have seen.
Things seem relatively normal compare to the measurement from August 17.
State of H1: locking DRMI1F but not able to get through Engage_DRMI_ASC - Sheila working on alignment by hand to get through this stage
Locking Struggles:
Site Activities:
Jenne, Stefan
- lowered CFSOF gain from 0.5 to 0.2. This avoids an instability due to gain peaking and possible radiation pressure feed-back at 2.15Hz.
- With that the coil driver switching was no longer an issue. In fact we reduced the ramp times all to 1 sec - it worked fine.
- We still keep dropping the sideband recycling gain and lost lock after ~36min. THus we decided to do some loop gain tracking.
- We added a laser frequency modulation line at 900Hz, and demodulated it in REFL_9_I (blue - LSC-LOCKINM_1_DEMOD_4_I) and REFL_45_I (brown - LSC-LOCKINM_1_DEMOD_3_I in the attached plot).
- Additionally we suspected that the BS ASC controls are going wacko. So we but pitch and yaw dither lines on the BS, and demodulated everything.
- Attached are two plots (with different time axis) of the result during a 50W lock.
Remarks:
- The REFL45 gain INCREASED by more than a factor of 2! This is consistent with less 45 sideband going into the cavity, and thus more being available as reference in REFL.
- The AS36_A gains seem to reflect that - i.e. a slight gain drop due to less 45MHz being available.
- The AS36_B gains initally agree, but then go completely nuts: AS36_B_Q_P (orange) keeps dropping, while AS36_B_Q_Y (red) turns around.
- Additionally, the AS36_B_Q_Y- Q dither demoduilation is growing significantly (not plotted). Not sure what to make of that. (the Q dither demodulation of all other signals remaind reasonable.)
Conclusion:
- We suspect that our lock losses might be due to the BS ASC signal becoming unsable. In addition, we have to carefully look at loop gains of all WFS that use 45MHz in REFL - their gain is expected to shoot way up.
- At lest for REFL ASC, it might be time to use the dynamic power normalization.
LEGEND for plot:
H1:LSC-LOCKIN_1_DEMOD_4_I_OUTPUT LSC: laser frequency to REFL_9_I (blue)
H1:LSC-POPAIR_B_RF18_I_NORM_MON 9MHz 2-f in PRC (green)
H1:LSC-LOCKIN_1_DEMOD_3_I_OUTPUT LSC: laser frequency to REFL_45_I (brown)
H1:ASC-ADS_PIT4_DEMOD_I_OUTPUT ASC: BS pit to AS36_A_Q_P (cyan)
H1:ASC-ADS_PIT5_DEMOD_I_OUTPUT ASC: BS pit to AS36_B_Q_P (orange)
H1:ASC-ADS_YAW4_DEMOD_I_OUTPUT ASC: BS yaw to AS36_A_Q_Y (purple)
H1:ASC-ADS_YAW5_DEMOD_I_OUTPUT ASC: BS yaw to AS36_B_Q_Y (red)
Beamsplitter motion as seen by its Optical Lever:
Over the course of the 40 minutes shown in Stefan's plots, the BS optical lever shows that it is moving:
I spent sume time lookin g at the signals form the AS36 diodes during this lock.
The 1st quation was whether all the RF from the length signals ios indeed in the I phase.
- Plot 1 shows the I sum and Q sum of AS_A_36. Most of the signal is in the I phase, but a phase change of -12deg (from -140 to -152 deg) would further improve the situation.
- Plot 2 shows the same thing for AS_B_36. It's overall phasing is as good as we can expect.
- Plot 3 shows the signal of all individual quadrants, I and Q (including the extra -12 deg in AS_A_36). Note that most signal is in quadrant 3. This might be an indication that we also have a centering issue, or it might be the result of an unhealthy about of higher order modes at the AS port.
- Plot 4 shows the Q signals that go into the BS error signals - they are a weighted sum of A and B - currently 0.666*A+1.0*B. For A the trace is plotted with (blue) and without (green) the extra -12deg of phase.
Note that the PIT signals (and to a lesser extend also the YAW signals) show a divergence to opposite direction, because their weighed sum is servod to zero. This is expected if one channel starts seeing more garbage than signal.
- Plot 5: This plot shows the BS dither line strength for PIT and YAW, for both A and B diode. Note that A is well behaved until the end, while B's gain increases significantly for yaw, and drops to almost zero for pit.
Based on plot 4 and 5 I suspect we would be better off using only AS_36_A at 50Watt. THe challenge will be it's offset - we will try this later tonight.
Last night, we ran a quick TCS test where we attempted to minimize the intensity noise coupling to the DCPDs by changing the CO2 differential heating.
It seems that the following CO2 setting gives a much better intensity noise coupling when the PSL power is 25 W:
This did not improve the recycling gain so much. It seems to have increased by 2% only.
According to a past measurement with a lower power PSL of 2 W (alog 26264), a good differential CO2 power had been found to be P_{co2x} - P_{co2y} = 270 mW (or probably less than 270 mW because I did not explore the lower differential power).
This could be an indication that ITMY has a larger absorption for the 1064 nm light such that the differential self-heating linearly changes as a function of the PSL power. We should confirm this hypothesis using the HWS signals.
[The test]
No second or third loops engaged, DC readout, no SRC1 ASC loop.
Drastic reduction of the intensity noise coupling was observed mostly between 4:24 and approximately 5:00, indicating that reducing the CO2Y power helped improved the coupling. After 5:00 UTC, we did not see a significant reduction. This may mean that we might have been already close to an optimum point where the coupling is minimized. The attached shows DARM spectra from various time during the test.
A broad peak at around 400 Hz is my intentional excitation to the first loop with band-passed gaussian noise in order to check the coupling to the DCPDs. As shown in the spectra, the reduction from the beginning to the end of the test is about a factor of 5. As reported in 27370, broad noise above 100 Hz up to several kHz is indeed intensity noise and therefore we see the noise floor in this frequency band decreasing too.
Because of the error in picomotor assignment, there's a good chance that the CO2Y laser is severly clipped and not properly aligned to the test mass. The result of this would be strong higher spatial order lensing (non-quadratic) on ITMY. We're certainly seeing an excess in lensing as measured with the HWS but the exact nature is unclear.
If there is signifiant higher order mode lensing then the best differential effect will be from having the CO2Y laser set to zero. However, this will not be the optimum lensing for the power recycling cavity.
So, poor CO2Y lensing is at least consistent with and a plausible explanation for requiring 0W on CO2Y to minimize intensity noise coupling while observing reduced PRC gain.
I looked back the intensity coupling of this particular day. See the attached.
The coupling at 100 Hz, even though the coherence is not high, seems to be too high by a factor of two or so comparing agaist the measurement from this February (25476). When dCO2 was sort of adjusted (red curve), the coupling at around 400 Hz and above seems comparable to what it was in this February.
