Times PST
9:58 Richard to EX to reconnect HEPI pump ground
10:15 Richard back
10:25 HFD on sit
11:20 Leo taking charge measurements
13:44 Joe D to both Mid stations
14:11 Joe D back
14:41 Richard to roof checking out GPS antenna work for tomorrow
Laser Status:
SysStat is good
Front End power is 32.64W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED
PMC:
It has been locked 6.0 days, 0.0 hr 31.0 minutes (should be days/weeks)
Reflected power is 2.464Watts and PowerSum = 25.33Watts.
FSS:
It has been locked for 0.0 days 0.0 h and 1.0 min (should be days/weeks)
TPD[V] = 1.643V (min 0.9V)
ISS:
The diffracted power is around 7.629% (should be 5-9%)
Last saturation event was 0.0 days 0.0 hours and 1.0 minutes ago (should be days/weeks)
J. Oberling, E. Merilh
On June 23rd we swapped the laser on the ETMx oplev, see alog 19290. We spent the next couple weeks tweaking the operating power of the laser to get it in a stable zone; this has to be done since the thermal environment is different between the end station VEA and the LSB lab the laser was originally stabilized in. After I got back from vacation last week I've been looking at quiet times (no apparent optic movement; I looked at the pitch & yaw oplev signals and picked times where the optics were quiet) to see if the laser is stable and glitch-free. I've attached 3 spectrograms of quiet times during the last week
There is no obivious glitching in the ETMx oplev laser as shown in these spectrograms. I think it is safe to say this oplev is healthy. I have also attached a spectrum of the ETMx oplev pitch & yaw signals for the same 4 hour stretch on 2015-7-19 as the attached spectrogram.
As usual if anyone notices anything not right with any of the optical levers, let me know.
Glitch Performance evaluation of diode lasers:
It is sometimes difficult to see whether a laser is glitch free because of the tools we employ display the data. It is good to compare performance of a laser under test with one that is known to be glitch free and another known to be glitchy. This way the display tool is validated (since it shows the glitches in the glitchy laser). At the same time we would know if the laser under test has achieved the Reference Laser's performance level.
Please see my post on laser evaluation. And it would be preferable if the same kind of plotting tools (ligodv) are used as in the past in order to make the comparisons easier.
This morning around 16:45 UTC, the ETMY ISI Stage 1 and 2 WatchDogs tripped reporting a "payload trip". Jim cleared these trips and then watched Guardian bring ISI back up in an odd way. Guardian brought the ST1 node to HIGH_ISOLATED but not all of the the preferred isolation loops on Stage1 were turned on, along with the input, output, and decimation. (Snipet of the log attached) Stage 2 proceeded to try and isolate but since stage1 was not entirely isolated the WatchDogs tripped again. This time, after clearing the WatchDogs, Guardian brought eveything back up properly and everything seems to have been working well since.
Jamie is unsure of the reason behind this, but suspects some epics connection issues.
Another odd bit to add is that the payload was never tripped...
At 17:40 UTC the ISC_DRMI Guardian had an the following error: "EzcaError: Could not get value from channel: <PV 'H1:LSC-PD_DOF_MTRX_SETTING_5_18', count=1, type=double, access=read/write>, None"
A caget of the channel yielded a value so it seems as though it was just the Guardian that was not seeing it. Reloading the code did not fix the error, and we were already on the phone with Jamie with another Guardian oddity (alog to follow). Jamie suggested that we STOP the node and then set it back to EXEC. This worked!
This has been seen before (example: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=19673) and seems to not be the same channels but from H1:LSC-PD_DOF_MTRX. Jamie and Dave both have their thinking caps on and are trying to solve the problem.
08:00-10:30 I got a little extra time for locking this morning.
Locking this morning was a little tricky as there was some FSS oscillation and ISS instability that was giving IMC a bit of trouble. Common Gain manipulation got that bit settled down and a bit of AOM diffracted power adjustment recitfied the remainder.
Here is a summary of the brute force coherence report already posted in a previous comment to an elog entry describing the good sensitivity lock of last Friday.
Basically, there is no large coherence anywhere, except for the well known periscope peaks that are coherent with ISS signals, IMC angular signals and PSL periscope (figure 1-3)
At low frequency, there is coherence with SUS-ETMY_L3_ESDAMON_?? signals. This was not there in the past, so I guess this coherence is just due to a change in the control strategy. If I'm not mistaken, this signal is just a monitor of the correction sent to the ESD, so coherence with DARM is normal. Please correct me if wrong... (figure 4)
In the 10-30 Hz there is coherence with ASC-MICH_P (figure 5)
In the 10-70 Hz region one dominant source of noise is longitudinal control, since there is coherence with MICH and SRCL (figures 6-7). This noise is not dominant and still a factor of few from the measured sensitivity.
In the higher frequency region (above 100 Hz), there is coherence with ISS and PSL periscope as already pointed out, but there is also some coherence with AS signal: ASC-AS_A/B_DC_SUM, ASC-AS_A_RF36_I_PIT/YAW etc... Together with the main jitter peaks, there is a broadband noise floor at about 1e-10 m/rHz from 100 to 1000 Hz. This might be intensity noise or noise in high order modes that is not completely filtered by the OMC (figure 8).
Finally, a 90 Hz bump seems to be coherent with HAM5 signals (figure 9)
SUS-ETMY_L3_ESDAMON_?? are the recently well-connected and well-digitized analog monitors of the ESD (i.e. TST or L3 stage) actuators. Since we're using ETMY L3 as most of our DARM actuator, it's no surprise that there is coherence with DARM below the DARM UGF. What's strange is that you post that they have coherence before they were connected correctly in the SUS-AUX model from whence they come (see LHO aLOG 19780) ...
Hang, Matt, Evan, Stefan, Rana
WE mostly were chasing weird locklosses and instabilities, but also managed to implement a few noise improvements:
Of the locklosses, some were just due to marginal points in the transitions and loop margins. The main issue over the past few days turned out to be that the TIDAL servo was turned OFF somehow on Friday evening. After switching that back on for ETMY, we have returned to having long locks. The highpassing of SRCLFF has removed the high power pitch instabilities that we were seeing.
We were able to get > 40 Mpc with 10W input. The butterfly mode of ETMY @ 6053.81 Hz is preventing us from turning on the OMC DC whitening right now, so we don't know how good our range really is, but our low frequency
We also got a change to reimplement a small amount of the ASC changes from before last maintenance day:
Next:
Note, Rana reports that the tidal was OFF at the far right switch on the IMC_F Filter Module (picture attached shows this switch now on as it should be).
Might be nothing, but the following alarms have been going off continuously for the past few days:
EX HEPI DIFF PRESSURE (yellow)
EX DUST1, EY DUST1 (RED), and LVEA DUST
The EndX HEPI Pressure Sensors are espectially noisy. EE promises efforts to quiet them. Meanwhile I've opened up the alarm levels to +-9psi from setpoint.
Matt, Hang We wrote some codes to optimize a2l coupling coeffecients. The script will measure omc dcpd's response to injected angular dither, and do a linear fit (for I and Q separately) of this response as a function of a2l gain. With a proper rotation of the I-Q plane s.t. the rotated Q' stays as a constant, the minimum of the response will be at the zero of the rotated I'. We ran the script for ITMX, ETMX, and ETMY. Results are shown in the attachments. The errorbars were underestimated because the raw data might be correlated if the sampling frequency was not low enough, thus we just plotted them as a reference but no minimization of chi-sq in the fit. Based on them, we did the following changes: H1:SUS-ITMX_L2_DRIVEALIGN_P2L_GAIN: 2.300 -> 1.947 H1:SUS-ITMX_L2_DRIVEALIGN_Y2L_GAIN: 1.350 -> 0.632 H1:SUS-ETMX_L2_DRIVEALIGN_P2L_GAIN: 1.100 -> 0.927 H1:SUS-ETMX_L2_DRIVEALIGN_Y2L_GAIN: 1.370 -> 0.927 H1:SUS-ETMY_L2_DRIVEALIGN_P2L_GAIN: -0.200 -> 0.000 H1:SUS-ETMY_L2_DRIVEALIGN_Y2L_GAIN: -0.600 -> -0.700
Seemed that our script worked...
Rana, Evan
Yesterday, Rana made a low-pass filter in order to remove EX ESD saturations when ALS DIFF is used as a DARM sensor. However, today we continued to see some locklosses around the time that DARM is handed off from ALS DIFF to AS45Q.
Therefore, we looked at FM8 and FM10 in the DARM filter bank, which are low-pass filters used with DIFF only. We found that we could relax the amount of rolloff in order to win some phase at 10 Hz and below (see attached foton plot). There is not nearly as much suppression above 500 Hz, but it doesn't really matter since most of the rms drive to the EX ESD is accumulated below 100 Hz. The attached spectrum shows the EX UIM and ESD drives both before (blue) and after (red) retuning. There is no increase in the the rms drive to the ESD (part of this is also due to Rana's retuning of the UIM/ESD crossover).
Finally, the attached transfer function shows the ALS DIFF OLTF before (blue) and after (red) retuning. The "before" OLTF was taken yesterday and has 4 dB less gain than the red transfer function (because we added 4 dB of missing gain in the LSC-DARM violin bandstop filters).
Additionally, we switched FM1 and FM7 in LSC-DARM. FM7 was a boost that would cause the OMC transmission to flutter when first engaged. FM1 was the suspension compensation, and since it is always on there is no point in having it in such prime real estate. So we've switched them and updated the appropriate settings in the ISC_LOCK and ALS_DIFF guardians.
We were also getting some transitions during this transition due to some unsuppressed 0.2 Hz microseism in the ESD drive. Ideally, all of the low frequency force is taken care of by the upper stages, but this was not the case here:
The new filters have a no-ramp integrator and a RG at 0.2 Hz which is Q-matched to the microseism bump. The L3 signal has much less 0.2 Hz signal and our ESD saturations are somewhat reduced. This seems to work fine on both ETMx and ETMy all through the low noise state.
Guardian scripts updated to reflect the new filters. Now the ETMX / ETMY switcheroo for flipping to low noise ESD state can be done by gain ramping alone -- no filter switching to match actuators.
Current version:
The previous upgrade to r1449 included a patch to throw errors if any other internal state inconsistencies (like that which caused the "double main()" bug) are detected. The errors encountered since Friday are from failures of these internal consistency checks, apparently from situations not fully covered in the battery of unit tests. This patch covers those situations. I'm looking into improvements to the unit tests to cover more of the user phase space.
No nodes have been restarted.
ISC_LOCK and ISC_DRMI were restarted around 2015-07-19 22:10:00 Z.
the probability a freeze-up does not impact at lease one dolphined FE is very small, so I'm using the h1boot dolphin node manager's logs to data mine these events. The dolphin manager was restarted when h1boot was rebooted Tuesday 7th July, so data epochs at that time.
As I was seeing with my monitoring programs, the restarts preferentially happen in the 20-40 minute block within the hour. The first histogram is the number of events within the hour, divided into 10 minute blocks.
We are also seeing more events recently, the second histogram shows number of events per day. The spike on Tue 14th is most probably due to front end computer reboots during maintenance. Friday's increase is not so easily explained.
FE IOC freeze up time listing:
controls on h1boot
grep "not reachable by ethernet" /var/log/dis_networkmgr.log |awk '{print $2r, $4}'|awk 'BEGIN{FS=":"}{print $1":"$2}'|sort -u
total events 197
minutes within the hour, divided into 10 min blocks
00-09 11 :*****
10-19 11 :*****
20-29 67 :*********************************
30-39 79 :****************************************
40-49 17 :*********
50-59 12 :******
events per day in July (start tue 07)
wed 08 09 :*****
thu 09 09 :*****
fri 10 08 :****
sat 11 07 :****
sun 12 08 :****
mon 13 09 :*****
tue 14 22 :***********
wed 15 10 :*****
thu 16 20 :**********
fri 17 38 :*******************
sat 18 16 :********
This is a very clever analysis, Dave I checked the LLO logs (there are three, corner, x-end, y-end). So far we only see these issues when we have a front-end down for IO chassis, new hardware installs.
Rana, Matt, Evan
Low-noise commissioning was derailed by various CARM-reduction locklosses:
During the ALS DIFF to DARM RF transitions, there were saturations of the ETM ESD actuators which caused occasional unlocks. This was due to the poor sensing noise in the ALS and so any change in the dark noise or EMI at the end stations would slow down the whole lock acq sequence,
We put the following low pass filter in DARM: started with the RED RLP80, but found the BLUE RLP180 was good enough so that the control signal is now dominated by low frequency stuff instead of the hash at 1 kHz. The script now turns this on by default and turns it off to recover the 100 Hz phase margin as soon as we have transitioned to a low noise DARM RF signal.
* also, while writing this, we had the second Guardian crash of the night using the latest build with the 'no double main()' fix. Jamie has been notified.
Today we've had >5 locklosses of the 3rd kind: where the CARM handoff from using arm transmission to REFL PDH fails. Attached is a lockloss plot of one of these events.
@ -1.2 seconds, CARM is on normalized REFL9_I
@ -1.3 seconds, it is starting the transition and the arm powers are starting to dive
Why are the arm powers diving here? I suspect that the normalized REFL signal may have some instability depending upon the L2A -> power coupling, so we might try freezing the normalization at that time.
Also, since we only normalize by Y arm power for CARM, we're getting a DARM-> CARM coupling at f < 0.1 Hz...
Cataloging the many ways in which we are breaking lock or failing to lock since Friday, we found this one:
Sitting quietly at 10W DC readout, there was a slow ring up of a pitch instability in several ASC signals. Perhaps its time we went back to seriously controlling the ASC in the hard/soft basis instead of the optic basis in which its currently done. The frequency is ~1 Hz and the time constant is ~1 min. It would be great if someone can look at the signs of the fluctuations in the OL and figure out if this was dHard or cHard or whatever.
In the attached plot, I've plotted the OpLev pit signals during the time of this ringup (0702 UTC on 7/19). The frequency is 1 Hz. It appears with the same sign and similar magnitudes in all TMs except ITMX (there's a little 1 Hz signal in ITMX, but much smaller).
Evan, Matt, Rana
We again saw the pitch instability tonight. We tried reducing it in a few ways, but the only successful way was to turn off the SRCL FF.
It appears that at higher powers, the SRCL_FF provides a feedback path for the pitch signals to get back to the Arms (since SRCL_FF drives the ITMs; and both of them as of Thursday). i.e. cSOFT has a secondary feedback path that includes some length<->angle couplings and produces a high Q unstable resonance. I don't understand how this works and I have never heard of this kind of instability before. But we repeatedly were able to see it ringup and ringdown by enabling SRCLFF.
To enable use of SRCL_FF, we've put a high pass filter into the SRCL_FF. This cuts off the SRCL_FF gain below a few Hz while preserving the phase above 10 Hz (where we want the subtraction to be effective). HP filter Bode plot attached.