On Tuesday, an upgrade was made to all ISI models to separate the WD untrip function from the Saturation Clearing function.  Also, an automated saturation bleed off was included to subtract accumulated saturations over time.  Saturations occuring in a given minute are subtracted from the total 60 minutes later.  This saturation has mainly been an issue for HAM6 when the shutter triggers and jars the ISI table.  While we can accumulate 8192 saturations on the actuators before tripping the ISI Watchdog, hundreds of these saturations can occur for each shutter trigger.

To prevent the ISI from ultimately tripping the ISI, the commissioners added a reset of the watchdog to the guardian.  Since Tuesday, the bleed off has been running but the guardian reset was also operational.  That is, until Sheila commented this out yesterday about 2200utc.  On the attached 24 hour trend of the saturation counter for the HAM6 actuator, I noted the ~time when the guardian reset feature was disabled and the one time when the new CLEAR SATURATIONS button on the watchdog medm was pressed.  You can clearly see the saturations that occur dropping off one hour later.  What is not clear is whether the clearing will happen quickly enough in a state when the system is dropping triggeringthe shutter frequently due to conditions or commissioning activities.  Still, at least here the conditions where such that the saturation were able to get back to zero.

TJ has been tasked with a SYS_DIAG addition to alarm/notify when the saturation total exceeds some percentage of the maximum prompting manual clearing to stop ISI tripping.

Summary:

I looked at all 10 events on Gabrielle's alog that are without "Those marked with ** look somwhat different in the spectral shape" caveat, i.e. I only picked the events that were similar according to Gabrielle.

We don't know what these are, we know that this is NOT a sensing side software/hardware glitch.

All of these look very similar (see attached). DCPD current suddenly shoots up, there's a msec-ish rise, and then an equally sudden and fast fall of msec order, then the servo slowly follows up.

Details:

The spikes for larger glitches are on the order of 10^-15m to 10^-14m, or 0.1%-ish RIN (0.02mA) (but there are smaller ones).

DC current increases, which means that X becomes shorter and/or Y becomes longer (if the IFO calibration is correct).

What we were able to exclude so far is:

• Any kind of sensing software/hardware glitches.
  • PDA and PDB are identical, DARM is identical to DCSUM sample to sample except a scaling factor, NULL doesn't show anything, and ASAIR_A_RF45_Q shows the same thing at the same timing which tells that RF45 thing was NOT the result of the DARM servo  imprinting the signal to DARM.

Things that are not yet completely killed are:

• Dust.
  • If it causes phase delay in Y arm, the sign of the thing is consistent with X-Y going negative.
  • The time scale of the thing is kind of right IF the dust is crossing the waist of the beam.
  • But the X arm dust glitches from June caused POP dc level and arm transmission to drop, which is not the case this time. In June there were events that caused the DCPD level to go up as well as down, which is not the case here.
• Test mass motion.
  • Bulk mass going in one direction and then returning to the original position in 1msec time scale is very unlikely as we need 3 impacts. First one, the second one that is twice as large as the first one in the opposite direction, then the third one that is the same with the first.
    • However, an impact on the mass followed by an excitation of resonance(s) might make it appear as a short glitch according to Gabrielle's experience with VIRGO (fascinating story about bursting bubbles in adhesives).
    • However, when Gabrielle looked at one event no significant excitation of anything was observed.
• TCS glitch.
  • If TCSX laser power drops, that would make MICH X length smaller, so the sign of the thing is consistent.
  • However, the case against the bulk mass motion also holds true here. The power should drop for 1msec, then the power should go up twice as large as the drop for 1msec, and then to the original power. It's not very likely.

Is there anything that helps us understand if this is happening inside or outside of the arms?

Plots:

These are four biggest glitches from Gabrielle's list but I looked at all 10 that are without "look somewhat different" remark.

On top is the calibrated DARM displacement, properly dewhitened, and rewhitened again to make the glitch visible while keeping the phase intact above 600Hz or so, and mildly band-stopped (-20dB) second violin resonances. The "calibration" of this is 10^-10 meters, and the sign is correct for msec signal (i.e. positive means X-Y going positive).

Middle is the OMC DCPD SUM. DARM UGF is about 40Hz (1/e time = 1/2/pi/40=4msec), so the glitch shape is almost undisturbed by the DARM servo.

Bottom (of the bottom) is the shape of the glitch after subtracting the background motion in OMC DCPD SUM.

J. Kissel, M. Wade

In order to test out the new GDS calibration pipeline infrastructure, Maddie needs for the newly installed EPICs values that represent the DARM loop model reference values at each calibration line to be populated. Since we don't have a new, vetted, model that represents the current interferometer yet, I've installed the ER7 values, which have been precalculated from Maddie. There are some SDF vs. EPICS values that still need sorting since some of the values are very, very small.

Below, I quote the values entered.
For the PCAL Line at 36.7:
H1:CAL-CS_TDEP_PCALY_LINE1_REF_A_REAL -6.6965e-17
H1:CAL-CS_TDEP_PCALY_LINE1_REF_A_IMAG 7.8784e-18
H1:CAL-CS_TDEP_PCALY_LINE1_REF_C_REAL 1.2887e+06
H1:CAL-CS_TDEP_PCALY_LINE1_REF_C_IMAG -193720
H1:CAL-CS_TDEP_PCALY_LINE1_REF_D_REAL 8.9995e+09
H1:CAL-CS_TDEP_PCALY_LINE1_REF_D_IMAG 1.1573e+10
H1:CAL-CS_TDEP_PCALY_LINE1_REF_C_NOCAVPOLE_REAL 0.999
H1:CAL-CS_TDEP_PCALY_LINE1_REF_C_NOCAVPOLE_IMAG -0.0462

For the DARM line at 37.3:
H1:CAL-CS_TDEP_ESD_LINE1_REF_A_REAL -6.4774e-17
H1:CAL-CS_TDEP_ESD_LINE1_REF_A_IMAG 7.6853e-18
H1:CAL-CS_TDEP_ESD_LINE1_REF_C_REAL 1.288e+06
H1:CAL-CS_TDEP_ESD_LINE1_REF_C_IMAG -196820
H1:CAL-CS_TDEP_ESD_LINE1_REF_D_REAL 9.1216e+09
H1:CAL-CS_TDEP_ESD_LINE1_REF_D_IMAG 1.1789e+10
H1:CAL-CS_TDEP_ESD_LINE1_REF_C_NOCAVPOLE_REAL 0.999
H1:CAL-CS_TDEP_ESD_LINE1_REF_C_NOCAVPOLE_IMAG -0.0469

For the PCAL line at 331.9:
H1:CAL-CS_TDEP_PCALY_LINE2_REF_A_REAL -8.1357e-19
H1:CAL-CS_TDEP_PCALY_LINE2_REF_A_IMAG -3.8605e-21
H1:CAL-CS_TDEP_PCALY_LINE2_REF_C_REAL 373720
H1:CAL-CS_TDEP_PCALY_LINE2_REF_C_IMAG -890150
H1:CAL-CS_TDEP_PCALY_LINE2_REF_D_REAL 7.7692e+10
H1:CAL-CS_TDEP_PCALY_LINE2_REF_D_IMAG -8.0292e+09
H1:CAL-CS_TDEP_PCALY_LINE2_REF_C_NOCAVPOLE_REAL 0.9206
H1:CAL-CS_TDEP_PCALY_LINE2_REF_C_NOCAVPOLE_IMAG -0.4128

More details to come.

Jamie, Dan, Gabriele, Evan

There are some files in evan.hall/Public/Templates/LSC/CARM/FrequencyCouplingAuto that will allow for automated transfer function measurements of frequency noise into DARM (using IOP channels) at specified intervals. Running ./launcher 1200 15 FreqCoupIOPrun.sh will open a dtt template, record the transfer function (and coherence) between the appropriate IOP channels, and then save a timestamped copy of said template. This then repeats every 20 minutes for 15 iterations (i.e., about 5 hours).

The idea here is to hook up the usual broadband, analog frequency excitation into the CARM error point and leave it running for the duration of the thermal tuning (1 V rms, 300 Hz HPF, 30 kHz LPF, with the CARM IN2 slider at -17 dB seems to be OK). This conclusion (i.e., to do a broadband frequency measurement only) was the conclusion that Gabriele and I came to after finding that a similar excitation injected into the ISS error point was highly nonstationary in the OMC DCPD sum.

If we want to do broadband noise injection to both the ISS and CARM, then some modification of the script will be required; i.e., we probably will have to ramp the two excitations on and off so that they are not injecting simultaneously. That's not hard, since both the CARM board and the ISS board have digitally controlled enable/disable switches.

Right now a thermal sweep with transfer function measurements does not really seem compatible with the heinous upconversion in DARM from the 508.289 Hz mode. Dan and I tried for some time to continue on with the work started by Jenne, Stefan, and Kiwamu, but we could not make progress either. We even tried (on Stefan's suggestion) sending in an awg excitation at the mode frequency into the EY ESD, but this did not have any real effect on the mode height. If it's useful, the EY mode #8 BLRMS filters are set up to measure the behavior of the violin modes in the neighborhood of this one.

Earlier today or yesterday, Sheila added some new "shuttered" states to the ALS guardians.  The ISC_LOCK guardian requests these states during the "bounce violin mode damping" state.  The new ALS states are just empty, so that we can have a final nominal state, so we know when we've completed everything.  This prevents the ALS guardians from reporting things like "lockloss" or "fault" when the green lasers are shuttered.  The new states only have a guardian edge coming from the "IR found" state. 

However, in the ISC_LOCK we were requesting these states, and then immediately closing the green shutters. If the ALS guardians detect that there is no green light in the arms before they've been told to go to this new state, they'll end up in "lockloss" or "fault".  I don't totally understand why, but if they are then told to go to the "shuttered" state from somewhere other than "IR found" , this causes a lockloss of the full IFO. 

Anyhow, for now I've put a 2 second sleep between the node requests and the closing of the green shutters.  At least one time we have now successfully made it through this state after my fix.  I will consult with Sheila and Jamie tomorrow about the best way to handle this - should we build in more explicit ways for the guardian to get to the "shuttered" state?

As a side note, we had several locklosses earlier in the day at this state, and I was unable to find any obvious reason for the lockloss.  Since these new states were already in use, I suspect that this problem is what caused those locklosses.

J. Kissel, D. Tuyenbayev, S. Karki, S. Kandhasamy

We've used the new infrastructure for the ESD / L3 / TST actuation strength tracking line for ETMY, turning on the line with a frequency of 35.9 [Hz] and amplitude of 0.08 [L3 LOCK ct]. We've tuned the excitation amplitude to be roughly half the amplitude of the 37.3 DARM and 36.7 PCAL lines, see attached ASD of DELTAL_CTRL for starters, and we'll iterate a bit once we get back to a better low-noise state (we're currently having a lot of trouble damping violin modes).

Since we were working on violin mode damping anyway (aLog 20280), we also tried some ETMX violin damping tuning.  At some point, we tripped the ETMX analog PUM current RMS watchdog.  We tried setting the H1:SUS-ETMX_BIO_L2_RMSRESET to zero, then back to one (which worked to reset the ETMY watchdog, when we tripped that once or twice).  We tried this reset procedure several times, but the watchdog isn't resetting. I don't know enough about this system to diagnose it further, so I await guidance from someone more knowledgeable in the morning.  Our low frequency noise is kind of terrible, which might be because it looks like we aren't driving the ETMX L2 stage at all, but we are able to get to the Nominal Low Noise state (the trip happened at the DC Readout state).

Attached is a screenshot of the current state of the ETMX suspension screen, including the tripped L2 RMS watchdog.

J. Kissel

I've updated several QUAD screens, including the overview, to reflect the changes that were made yesterday. In the process, I've also added some stuff to the overview screen that was previously only shown on sub-sub screens such that problems are better exposed from the top level. I attach screenshots, but the changes include:
SUS_CUST_QUAD_OVERVIEW.adl --
- Adding a link to a new screen for the new ESD / L3 / TST calibration line.
- Cleaned up the LHO-only, UIM tidal path, better showing how it's added into the UIM control signal.
- Brought the HV ESD driver ON/OFF switch to the overview, and renamed it from the misleading "RESET" to "ON/OFF."
- Brought all relevant voltage (for the ESD) and current (for the OSEMs / coils) up to the top near the output.
- Added a red light surrounding the COILOUTF block that appears if all of the OSEM sensor signals are zero (literally an AND of all four, checking whether the INMONs are < 5 cts).

SUS_CUST_QUAD_BIO.adl --
- Changed the HV ESD driver name from the misleading "RESET" to "ON/OFF."

SUS_CUST_QUAD_L3_ESD_LIN_WITH_CHARGE.adl --
- Added a whole bunch of text and polygons all over the screen to better indicated what "linearization bypass ON vs. OFF" means, since this switch confuses everyone.

SUS_CUST_QUAD_MONITOR_OVERVIEW.adl
- Removed now defunct EPICs records and level indicator bars for the DIGITAL readbacks of the ESD HV driver, which are now just binary bits.


All changes have been commited to the userapps repo. To inherent at LLO, just svn update
/opt/rtcds/userapps/release/sus/common/medm/quad/
the directory.

Related alogs: 17610, 19720

Jenne, Stefan

We had highly rung up violing modes tonight, in particular 508.289Hz was outrageous.
Nominally it should be on ETMY. We pushed on that optic in pitch, yaw, length, and with all sorts of different filters (narrow ones, broad ones, different feed-back phases, etc.)
We succeeded in ringing up all three neighbouring modes, and thendam them again. But 508.289 never changed its size. We even tried pushing ETMX (its modes are closest to ETMY), but no success...

(There was a typo in the frequency in the first version, hence Evans comment)

Current state of ASC sensing and actuation.

[Note: The INP2 loops are not used during lock.]

Charge measurements done on both ETMs. Measurements results are in attachments. The same trend remains (discussed in 20067)

Patrick, Carlos

We disabled and stopped the following services on h1ecatx1:

Acrobat Reader Updater
Windows Update
Windows Search
Windows Defender

In all the last lock stretches we saw a lot of very loud glitches, clearly visible as large drops in the range. See reports from Sheila and Lisa.

I had a look at the time period pointed out by Lisa (1122456180 - 1122486360). I wrote a MATLAB script (attached) that load the range and select the glitch times looking at the large drops. Since the range is sampled once per minute, I then look at the DARM error point and search for an excess of noise in the 30 to 300 Hz region. In this way, 16 loud glitches were selected.

Most of them are similar, see the first figure. I checked with the MATLAB script the value of all suspension actuation signal, using the MASTER channels. No saturation or unusual behavior is visible.

I also checked that there is no drop or change in the IFO powers. Then, as suggested by Sheila, I checked the DACKILL signals as well as the CPU_METER signals. Nothing there.

Finally, I selected the loudest glitch (1122479776) and looked at all (literally) *_DQ channels. I couldn't find any interesting correlation, except for signals that are known to be related to the DARM loop, like for example ETM* and ITM* control signals. For this glitch and a couple of the loudest ones I could see a similar glitch in H1:LSC-ASAIR_A_RF45_Q_ERR_DQ, H1:ASC-AS_A_RF45_I_YAW_OUT_DQ, H1:ASC-X_TR_A_NSUM_OUT_DQ, H1:ASC-Y_TR_A_NSUM_OUT_DQ. All these are signals somehow sensitive to DARM.

In conclusion, I couldn't find any channel, except DARM ones, that are correlated with those glitches.

Rana pointed out to me that the PR3 and SR3 suspensions may still have some shift due to wire heating during locks (which we won't see until a lockloss, since we control the angles of mirrors during lock).

Attached are the oplev signals for PR3 and SR3 at the end of a few different lock stretches, labeled by the time of the end of the lock. The lock ending 3 Aug was 14+ hours.  The lock ending 31 July was 10+ hours.  The lock ending 23 July was 5+ hours.  The lock ending 20 July was 6+ hours.

The PR3 shift is more significant than the SR3 shift, but that shouldn't be too surprising, since there is more power in the PRC than the SRC so there is going to be more scattered light around PR3. Also, PR3 has some ASC feedback to keep the pointing.  SR3 does not have ASC feedback, but it does have a DC-coupled optical lever.  SR3 shifts usually a few tenths of a microradian, but PR3 is often one or more microradians.  Interestingly, the PR3 shift is larger for medium length locks (1 or 1.5 urad) than for very long locks (0.3 urad).  I'm not at all sure why this is.

This is not the end of the world for us right now, since we won't be increasing the laser power for O1, however we expect that this drift will increase as we increase the laser power, so we may need to consider adding even more baffling to the recycling cavity folding mirrors during some future vent.

[Matt, Jenne, Evan, Sheila]

There is an enormous peak in the DARM spectrum at 4735 Hz.  Shown in the DTT printout below is the IOP channel for the OMC DC PD (H1:IOP-LSC0_MADC0_TP_CH12), from 1 kHz to 25 kHz, and this 4.7 kHz peak is dominating by about 2 orders of magnitude.  

We wonder if this is perhaps an acoustic internal mode of one of the test masses, although we are having trouble finding a listing of such modes.  

Does anyone know where we can find a listing of test mass acoustic modes?  Or, alternatively, does anyone have any thoughts on what this mode might be?