Please when the laser trips, disable the locking for the PMC, FSS, and ISS. It doesn't make much sense to keep ramping the NPRO PZT when there's no chance of any of the cavities acquiring lock.
CDS: Continuation of framewriter debugging Tues: reinstall common mode board SEI: Tues: offload BS HEPI DC bias to large springs VAC: h0vaclx restarted to fix annulus ion pump channels. Richard reset ITM ESD high voltage. Kyle turning off pumps
When CO2Y laser had an issue (alog2856) I added a 1 degree offset to the rotation stage calibration to trick it to deliver the right amount of power. That offset has been removed and the CO2Y RS now delivers the right power again.
For higher accuracy the RS random walk script has to be run.
Attached is a capture of the PSL's TwinCAT status screen. Same problem as previously, namely a flow error in the
AMP cooling circuit.
In bring the laser back up, I took the opportunity to reboot h1pslctrl0. This has temporarily fixed the problem
with the diode current readout of diode box 3.
Attached are zoomed in plots of the flow rates and pressures. Note an oscillation is present in the flow rate
signals, which might be indicative of something "flapping" in the water flow. A plot of the laser head flow
rates is also attached. The flow through the laser heads is not the problem, at least not this time.
In looking at the pressure fluctuations, it looks like they start in the inlet side before appearing
in the outlet side. Both have appear to have a period of 1s, although this might be an artifact of the
sampling/recording of the trend data.
A quick visual inspection of the filters in the chiller room didn't reveal any new debris.
[Sheila, Jenne, Kiwamu, Terra]
We have reset the initial alignment offsets, including realigning several paths on ISCT1, to match the alignment that we liked from yesterday (alog 29430). So far, we think that it's good. We have powered up part way twice since, and the power recycling gain no longer drops very much! It still drops a bit, so we could consider going a bit farther, but this is probably plenty good to move on and focus entirely on low noise efforts. Now at 40W it is something like 31, rather than something in the mid/low twenties. Plots coming tomorrow.
In order to do this, we did the following (hardware alignment changes noted in bold):
1st screenshot: powering up from 2 watts to 32 Watts without loosing recycling gain. The POP18 build up still drops, but AS90 is much more stable than with previous alignments. According to the POP/IM4 monitor, this is a recycling gain of over 32 at 32 Watts, calculated from the arm transmissions it is a recycling gain of 34.4
2nd screenshot: Slider values for new alignment.
3rd screenshot: Modification to CSOFT pit filter to add phase and gain around 0.4 Hz, the frequency of the dPdtheta instabilty. On the right side of that screenshot is a measurement of the loop, taken with the digital gain set to 0.1. The nominal gain used for the references was a factor of 5 higher and we have now restored the loop to that gain. We can probably get more supression of the instability out of this loop.
Apart from the recycling gain, we seem to see a slight improvement in the beam shape at the dark port after the PR2 move. The below are camera images of the AS port a few days ago and today, respectively
A concentric circle pattern (with circle pattern's center at around x,y = 350,270) became dimmer. Note that the old image was taken about 17 min. after the interferometer had reached 50 W, while the latest image was taken a few minutes after a 50 W lock. Also, the third attachment shows a comparison of the horizontal beam profile around the center in which the latest picture shows less significant features.
8:13 UTC
TITLE: 09/02 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Sep
SHIFT SUMMARY: I arrived at the tail end of the 7.2 EQ in New Zealand, so commissioning efforts were just starting again. They continued throughout the night, working hard as always.
On Tuesday we reconfigured h1fw1 to run the newer code (same code as h1fw0), and put the older code on h1fw2 for comparison purposes. So: h1fw0 and h1fw1 are running r4252 from branch-3.0 in svn. h1fw2 is running as the advLigoRTS-3.0.3 release tagged for O1. We have not seen frame writer crashes in several weeks now. However we are seeing h1fw0 produce a small number of frames that are different from what is produced by h1fw1 and h1fw2. With analysis done by Greg Mendell and David Barker it appears that the daqd does not receive all the data occasionally and issues some re transmit requests for some of the data. When it does this some of the data is put in out of order. So apparently data that should arrive in as 1 2 3 4 5 may be stored as: 1 3 4 5 2 I will review the network receive code tomorrow to see if this is what is happening. We find it curious that it inserts the data it skips later on. We are not sure what is causing this, but a leading theory is that this may be a network issue between the h1dc0 and h1fw0. Next Tuesday Dave plans to switch the connections on the switch between h1fw0 and h1fw1. If the behavior switches between h1fw0 and h1fw1 we will be able to attribute this to a problem with either the port, sfp, or fiber that is currently connecting h1fw0 to the switch. I will work with Carlos to see if we can get error counters from the switch.
The plot shows that an alignment study last night changed jitter coupling reproducibly. Notice that the different peaks from the PSL seem to change differently, so it seems more complex than something like clipping.
Is this really incompatible with jitter on an internally badly aligned PD array?
Elli (remotely), Kiwamu,
Today was another day for the SRC gouy phase measurement. Even though I said that I would try out another misalignment configuration (29418), we decided to repeat the same measuremnet/configuration as yesterday with an improved clipping situation.
We will stop doing the measurement for now until we hear from Elli on her offline analysis. Done for now.
[The measurements]
[The setup]
[The data]
All the relevant data is available at kiwamu.izumi/Public/measurements/20160831_SRCgouy2/data2
Still thinking about earthquakes, I have 2 main points:
1. We can reduce the number of times ISIs trip during earthquakes by reducing the amount of inertial isolation we do a low frequency. Even if we can't lock, changing seismic configurations may save time in recovery.
2. The current earthquake configuration amplifies microseism (.1-.3hz motion) by a factor of a couple. I have some thoughts on improving that, but we need to balance not making things worse at .1-.3 hz (for the summer, we'll need to do better come November) and rolling off the seismometers quickly below .1hz. I've made a Earth_Quake_V2 to address this, but theres more work here to do.
This morning New Zealand had a pretty big earthquake, with peak .03-.1hz blrms velocties around 10 micron/s at LHO. We noticed this before the velocities reached around 1 micron/s, so I moved the SEI_CONF manager to the Earth_Quake state. Looking at ground and WD trends for the ISI's over the last two months, it seems that normally the BSC ISIs trip at 1-2 micron/s, and today all of the platforms survived. My first plot shows the ITMY .03-.1hz BLRMS, ITMY ST2 WD MON bit and SEI_CONF state (40 is the "windy" state, 17 is the "earthquake" state) for the earthquake today, the second is a plot of an earthquake from Aug 29th. The earthquake on Aug 29th took out all of test mass ISIs (WD_MON goes from 1 to 4), while none of the chambers tripped today, even though the peak velocities were almost 1.5x as high. Kiwamu was even able to run his Guoy phase measurements, without losing the IMC, or any other platforms. He did this with when the .03-.1hz blrms were sitting around 10 micron/s.
Given the fact that we can't lock with 1+ micron/s, I think it would be relatively low risk to automate switching the seismic configuration, for the case where some number of site seismometers are showing 1 micron/s motion. Maybe we could add this to the SEI_CONF manager.
I really doubt we could get to NLN or even ALS during an eq like this and this may be harder to do when the microseism is high. I think this shows that even if we can't lock the IFO, it might still be worthwhile going to an earthquake robust state to reduce the amount of re-alignment needed to recover, once the earthquake is over.
I've been looking at data from the earthquake on Aug 31, in alog 29403. My third figure is the stock lockloss plot from the first lockloss during that eq. Looking at ALS in the "earthquake" and "windy" seismic configurations on Sheila's plot in 29403, ALS sees more .15ish hz noise in the earthquake state. This is because the current earthquake state turns off the low frequency sensor correction and so the .1-.3hz motion gets amplified by the gain peaking of the 250mhz blends on the ISIs. On the BSCs this gets amplified twice, because both stages use 250mhz blends.
One way to get around that is to use some narrow band sensor correction to reduce the .1-.3hz motion. I've tried this a bit during the earthquake today, and it seemed like it could have been helping, but things were obviously moving a lot. For now, I've added and "Earth_Quake_V2" state to the SEI_CONF that adds .1-.3 sensor correction to the earthquake state. Commissioners and operators should try using this state the next time we get a ~.5micron/s earthquake.
Had to trend and move back PR2 alignment sliders to be able to find IR. After that I had trouble locking DRMI and PRMI so I ran through an initial alignment. No trouble after that. Kiwamu ran his Gouy phase measurements in the morning during which time there was a 7.1 mag. earthquake off the coast of New Zealand. The PSL also tripped during his measurements and Peter brought it back. I updated h0vaclx and h0vacly to add beam tube annulus ion pumps. John W. turned up the air conditioning in the control room. Kyle has been periodically turning down the heating on the HAM4 RGA bakeout. Terra has been taking PI measurements. Jenne and Terra are now recovering the IFO from a lock loss. 15:48 UTC Karen opening OSB receiving door 15:54 UTC Chandra to HAM4 to check on RGA bakeout 15:59 UTC Kyle to HAM4 RGA to turn down heating 16:02 UTC Chandra back 16:06 UTC Kiwamu starting Gouy phase measurement. Setting IFO to down. 16:09 UTC Kyle back 16:22 UTC Filiberto to end Y to check on Beckhoff end station 3 chassis 16:42 UTC PSL tripped off, Peter investigating 16:53 UTC Travis, Betsy, Mitchell to LVEA for 3rd IFO inventory 17:15 UTC Richard checking on LVEA access door 17:19 UTC Richard done 17:21 UTC Kiwamu to ISCT6 17:54 UTC Kyle to HAM4 RGA to turn down heating 17:58 UTC Kiwamu back 17:58 UTC Kyle back 18:07 UTC Gerardo to LVEA to check on new beam tube annulus ion pumps 18:34 UTC Travis, Betsy, Mitchell out 19:07 UTC Filiberto heading back 19:21 UTC Kiwamu to ISCT6 to dismantle setup used for Gouy phase measurement 19:29 UTC Kiwamu back, done with Gouy phase measurement 19:37 UTC DAQ restart to add vacuum channels for beam tube annulus ion pumps 19:37 UTC Karen to OSB optics lab to clean 19:44 UTC Filiberto turned back on ITM HV (tripped off from vacuum code restart) 20:09 UTC Karen out of optics lab 20:14 UTC Betsy to LVEA west bay to check serial numbers on parts 20:19 UTC Chandra to mid Y 20:22 UTC Kyle to HAM4 RGA to turn down heating 20:23 UTC Travis to LVEA west bay to help Betsy 20:31 UTC Gerardo to LVEA to check signals of beam tube annulus ion pumps 20:33 UTC Kyle done 20:40 UTC Started initial alignment 21:26 UTC Completed initial alignment 21:34 UTC Travis and Betsy done 22:24 UTC Kyle to mid Y 22:25 UTC Jenne and Sheila to ISCT1 22:44 UTC Jenne and Sheila back
WP 6132 I updated the code from svn to add the new beam tube annulus ion pumps (LX IBM 122, LX OBM 136, LY IBM 192, LY OBM 196). It turned out that I connected the LX annulus ion pumps to the old PT110 channels instead of the old PT170 channels and thus the readings for these are currently wrong (the signal cables were landed on the old PT170 channels). This has been fixed in svn but h0vaclx has to be updated and restarted again at some later opportunity. As expected the restart tripped the ITM ESD high voltages. Filiberto reset them. I burtrestored both IOCs to 8 am. Dave added the channels to the DAQ. I'm leaving the work permit open until h0vaclx is updated and restarted again.
Correction on Sep 1:
Some zero/pole numbers in the original entry was incorrect, so I made some corrections below.
Specifically, ILPD has zp=([71mHz;72mHz;2.7k],[3.1;3.4;130;2.3k]) (130 and 2.7k were swapped in the original entry). Plots were updated accordingly.
Summary:
What's done and found:
These things were mostly done during Tuesday maintenance.
Some more explanation:
Since the outer loop injects its signal to the inner loop error point, OLTF of outer loop is basically the ratio of two transfer functions, one from RIN to the output of the outer loop servo board (H_OLBoard) and the other from RIN to the inner loop error point (H_ILPD):
OLTF_OL=H_OLBoard/H_ILPD.
As I noted before (alog 26879 and an entry attached later to that one), the inner loop diode is equivalent of 660 Ohm DC trans impedance and zp=([71mHz;72mHz;2.7k],[3.1;3.4;130;2.3k]) . DC output of this inner loop diode is about 1.6V differential (measured). For details you need to look at D1001998.
H_IL = 1.6[V]*zp([71mHz;72mHz;2.7k],[3.1;3.4;130;2.3k]).
OTOH, outer loop trans impedance amplifier doesn't have any coloring in the new prototype (the old one has this compensation, see D1300639), and each diode outputs about 130mV or so at 2W. There are four PDs, so the sum of all PDs at DC is about 520mV at 2W. The outer loop chassis has a flat gain of 3, a variable gain (VGA) with -12.8 to 45.6dB range, and a servo filter SERVO=zp([600;9k],[20,20,380]) with the DC gain of 1 (see alog 29293). (There should be another factor of 2 from SE to DIFF conversion, but the negative leg of DIFF is grounded, so this is not there at the moment.). There's also a MC pole (9k) in H_OLBoard.
H_OLBoard(2W) = 0.52[V]*3*IMC*VGA*SERVO = RIN*1.6[V]*VGA*zp([600],[20,20,380]).
Combining these,
OLTF_OL @2W = H_OLBoard@2W/H_ILPD ~ zp([71mHz;72mHz;2.7k],[3.1;3.4;130;2.3k])/zp(600, [20;20;380])/VGA.
Thing is, H_IL is absolutely huge at 3Hz and higher due to double 70mHz zeros, and the outer loop board as of now doesn't have the gain necessary to keep the UGF at some kHz.
Second attachment bottom shows the above theoretical OLTF at 2W with the maximum (45.6dB) VGA and the measured data. Remember that the "measured" is no good measurement because it was saturating, but nevertheless you can see that the measurement is not totally unreasonable.
It's clear to that we need something in the outer loop to counter 70mHz double zeros.
Implementing that would also help AC coupling. As of now, the AC coupling is already sort of aggressive, but not aggressive enough. If I bring up the blue curve in the second plot bottom by 80dB or so to set the outer loop higher UGF at 2kHz, the lower UGF would become .2Hz or so, that might not be good enough of an AC coupling.
Kyle, Chandra Starting step down of heating of Vertex RGA -> Closed calibration gas bottle isolation valves and reduced variacs (except one supplying turbo inlet) by 10% -> Will need access to continue this process every few hours or so throughout the day.
~1100 - 1110 hrs. local -> Kyle in and out of LVEA -> Reduced variacs 10%
~1325 - 1335 hrs local -> Kyle in and out of LVEA -> Closed 1 1/2" turbo isolation valve, reduced variacs 10%
1455 - 1300 hrs. local -> Kyle in and out of LVEA -> De-energized variacs, i.e. killed heat to Vertex RGA
Unsuccessful at damping ITMX 15520 Hz PI several times tonight (previously seen here and here). We find that larger damping drive does not equal greater damping: When this mode was test driven and damped after the thermal transient at 50 W, a best gain and phase was found for damping. When the mode began to ring up later, increasing gain (by some large amound but still under saturation) or flipping sign and/or changing phase only resulted in faster ringup. This is true when still far below DAC saturation levels. It seems as if there is some gain sweet spot that must be found.
--- --- ---
We had three occasions to damp ITMX 15520 Hz mode during the night. During the first, I successfully damped and it then rerang up (perhaps due to offset adjustments?) and lost lock. During the second and third I was not able to damp the mode and avoided lockloss only by decreasing power 50 W --> 25 W.
Below you see the mode first ring up and my gain trial and error response until I settle at 10000 and the mode is fully damped. Soon after, you see the mode ring up again right after the yaw offset step from ~ -0.02 --> -0.01. Note we started with a small negative gain so I just assumed we had actually been ringing up the mode the past few days.
During the second 50 W lock, damping was already running at the gain and phase settings that were effective at damping the first ring up above (+10000 gain, -60deg). Despite this, you see the mode slowly rising ~3 hours into the 50 W lock and my gain adjustments trying to damp. Note that here I start with some mostly successful positive gain (i.e. shallow slope) yet both raising the gain and flipping the gain sign cause the mode to ring up. I tried phase changes at this point too but existing was best. I avoided lockloss by decreasing power to 25 W, allowing mode to ring down enough to damp, then powering back up. I also rechecked my gain and phase at 50 W (post thermal transient time) and it would still drive and damp with a very steep slope. Still, an hourish later the mode began to ring up. I found similar behavior in the third attempt as the second and had to decrease power to avoid lockloss.
Things to note:
Things to try:
I've plotted the HOM spacing (H1:TCS-SIM_IFO_XARM_HOM_SPACING_HZ_OUTPUT) from the TCS simulator vs the RMS of the 15520Hz PI mode. It seems to be ringing up consistently when the simulated HOM spacing edges up over 5034Hz.
The first plot shows the HOM spacing at the same time that Terra sees and tries to damp the mode. You can see the HOM spacing edge up over three hours as the surface curvature is becoming flatter. The 15520 mode starts to ring up and then Terra is able to damp it. It looks like the subsequent yaw offset increased the power in the arm very slightly which has, in turn, increased the heating of the optic. The estimated HOM spacing increases, most likely increasing the parametric gain of the 15520Hz mode in the process.
The second plot shows the HOM spacing over a larger time frame (19 hours) and the associated RMS of the 15520Hz mode. Every time the HOM spacing reaches 5034Hz, the mode starts to ring up.
Some notes:
In fact, it's not too much of a stretch to use the parametric gain of the modes in conjunction with the simulated HOM spacing to continually update the total absorbed power in the arms.
Thanks Aidan.
I've attached a look at the HOM spacing during two times that this same mode rang up while no damping was being applied (DAMP_GAIN = 0), unlike the times you looked at. First time the mode rang up when HOM spacing was about the same as you found, 5035. Second (indicated with red arrow), rang up around 5025. Both locks were 50 W.
The new prototype board was modified to satisfy some of the things in the requirement (T1600064) as listed in 1., 3. and 5. below. I also listed things that were not modified but would have been good if recommendations in T1600064 were implemented.
I wouldn't claim that we won't modify it further, but this is just the current state.
In this entry you'll be able to see the electronics modifications in the attached, but the photos are too small to read the original component values, so read the original drawing D1600298 as necessary.
1. Readback of the PD array signal after the servo loop switch ("SUM PE MON") needed to be DC coupled.
The "whitening" was originally AC-coupled (second attachment) and therefore it was not compatible with digital AC coupling described in the requirement.
We bypassed the big caps in the input of the opamps with some resistors so it acts as one single pole at 2.7k with DC gain of 50 (second attachment, mod 1).
To make the OLTF measurement easier, we made the same change to the whitening downstream of the summation for the excitation DAC output.
The servo board output readback was also AC-coupled, but we want to see if everything is working fine including DC injected into the inner loop board.
We made the output whitening a true whitening (second attachment, mod 2). 0.35Hz zero might be too low, though.
2. The board doesn't implement the analog compensation for the in the inner loop filtering upstream of the error point.
The inner loop error point "whitening" works as a very steep boost seen from the outer loop, roughly an equivalent of (z, p) = ([3; 3; 130], [0.07, 0.07]). T1600064 reccomends to implement the same thing in the outer loop somewhere to cancel this effect to make the loop design simpler.
Since this is not compensated in the prototype board, the outer loop OLTF would become huge for f<3Hz. This means that the digital AC coupling servo should work REALLY hard to effectively AC-couple the entire outer loop at, say, f<1Hz, without reducing the outer loop gain at 10Hz and up.
We do not implement this by analog cut and paste job (no opeamp available for this on the board), but we'll try to make it work by an aggressive digital AC-coupling filter.
3. Integrator as a boost doesn't go well with digital AC coupling so it was converted to a usual boost with finite DC gain.
The error point of the digital AC coupling servo is kept zero at DC (i.e. a pole at zero Hz). The outer loop prototype had an integrator as a boost.
Of course they don't go well as any tiny DC difference between the AC coupling error point and the integrator input will be integrated and eventually the servo runs away. But we don't really need a pole at zero Hz.
We changed it such that the first boost acts as 50Hz LPF with DC gain of 10, and when added to a flat unity gain it becomes 500:50 boost (third attachment) (but see the next entry).
4. Boosts were implemented as three parallel integrators added to a flat unity gain.
We had a choice of unity gain, zp=50:0, 100:0 or 150:0. See the first and the third attachment.
As described in 3. above, we already changed one integrator to []:50 LPF so the boost becomes 500:50.
We didn't fix the parallel summation, it might be OK to go without any boost or using just one.
But T1600064 shows our standard practice of connecting boosts in series.
5. Outer loop servo filters on the board were also changed.
Originally the servo filtering was something like (z,p)=(100, [20^2;40]) (fourth attachment). As described in 2. above, 1st loop effectively adds ([3^2; 130], [0.07^2]). IMC adds another pole at ~8kHz.
Everything added together it's like ([3^2; 100; 130], [0.07^2; 20^2; 40; 8k]), and this might be OK.
But we made a change so the board became ([380;9k],[20^2;600]) by changing one of zero-Ohm resistors in the second stage and one C-R pair in the third stage (fourth attachment). Everything added together it would be ([3^2;130;380;9k], [0.07^2; 20^2;600; 8k]).
This change might not have been necessary. We just felt that letting a big 4.7uF capacitor and a zero-Ohm resistor to determine AC gain at around 10kHz in the first as well as the second stage was maybe too much of an uncertainty in the AC gain.
6. Sallen-Key at 0.1Hz for Digital AC coupling path might be too aggressive
For the moment the Sallen-Key is disabled for the digital AC coupling path (so the only analog filtering is a 10Hz LPF) to make things easier as we try to make it work.
Outer loop and digital AC coupling topology
Error point for the digital AC coupling is upstream of the summation point for the outer loop and the third loop (ERR1 in the attached medm). This comes from two requirements:
If the error point is downstream, the 3rd loop gain will be eaten by the AC coupling loop.
This means that, as we change the VGA gain, the offset coming between the AC coupling error point and the input to the VGA will be amplified and injected downstream, and if this becomes the problem we need to compensate for that using the analog excitation offset.
Second attachment shows the AC coupling OLTF, with the setting shown in the first attachment.
I was looking (again) at the ISS inner loop circuit diagram to make sure that I understand the system correctly and found that my alog 26879 was wrong about inner loop diode in that one pole and one zero were somehow swapped (see the new entry I've just made as an attachment to the alog 26879.
Anyway, since the amplitude actuation of the outer loop summation point of the inner loop is the inverse of the inner loop sensing "whitening", 2. should read
"The inner loop error point "whitening" works as a very steep boost seen from the outer loop, roughly an equivalent of (z, p) = ([3; 3; 2.34k;2.7k], [0.07;0.07;130])".
Sorry for a confusion, it seems like it's too late to fix the original alog entry above.
Correction of correction (sorry).
Original alog 26879 seems to have been correct, therefore the origial entry of this alog should be correct.
The inner loop error point "whitening" works as a very steep boost seen from the outer loop, roughly an equivalent of (z,p)=([3; 3; 130], [0.07, 0.07])
Sorry for further confusion. Again I cannot correct the above original entry nor the above correction any more, thus this entry.
It would be best to write a guardian to do this if it is necessary. There are not always people here when the laser trips.
I added a few lines to the LASER_PWR Guardian so that when this node goes to FAULT from a lack of light, then it will turn OFF the autolockers for the FSS, PMC, ISS in that order.