late entry for yesterday's Tuesday Maintenance

SUS PI

Carl, Dave, Terra, TJ:

new models for h1susetm[x,y]pi with associated DAQ restart.

Terra and TJ got the new Guardian GRD_SUS_PI node started.

New Vacuum Controls PID code for CP7

Patrick, Dave:

New code and epics database installed on h0vacey. New INI file installed with associated DAQ restart. New autoBurt.req file installed in target area.

New ASC model

Jennie, Jim, Dave

New h1asc model installed. After the build process we saw confusing archived INI files. We rebuilt, restarted and performed a DAQ restart.

New OAF model:

Jeff, Dave

A new h1oaf model was installed. Surprisingly the CPU processing time went up by 3uS from 53uS to 56uS. This is getting uncomfortably close to its 60uS limit. The changes made yesterday should not have done this. More investigation is needed. DAQ restart was not needed as this was a very unintrusive model change.

New dataviewer code

Jim:

Installed new dataviewer code with bug fix to trend the new double precision floating point data type.

new NDS2 client code

Jim:

version 12.2 nds2 client code installed on CDS.

mini-epics-freeze events

Sheila, Dave:

Sheila reported and alogged some epics freeze up events while viewing ASC and LSC data. I did a quick analysis using my IOP duotone channels which get to the DAQ via channel access and did not see any freeze-up in these channels for ASC and LSC. This appears to be a new phenomenon, investigation is continuing.

runtime, graphical

For a running system a filter's SWSTAT value and its decoded switch bits can be viewed graphically by pressing the "GUARDIAN SET" button on the lower left of the filter medm. Of course these bits are also shown on the main medm, the FILTALH medm just shows just the switch settings more clearly.

command line, non graphical

For a command line decoding of the bits, Jamie has written the 'cdsutils sfm decode' function. This takes either one argument (the SWSTAT value) or two arguments (either SW(1,2) or SW(1,2)R). For a running system you can use either SWSTAT or the SW(1,2)(S,R) options. For past settings only SWSTAT is available from the DAQ. From conlog the SW(1,2)S values are also available, but they are equivalent to SWSTAT (filter engaged bits not available). Here is an example with SW(1,2)R supplied (so engaged bit data is shown).

cdsutils sfm decode 12292 1536

INPUT
FM5
FM5_ENGAGED
OUTPUT
DECIMATION

command line, graphical

Mike Thomas has written a python script called filterstate, which takes either SW(1,2)S or SW(1,2)R pairs as arguments and shows the switch settings graphically by creating a static png image which approximates to the standard filter module medm. Because only SWSTAT is available from the DAQ. I have written a script to convert SWSTAT to its equivalent SW(1,2)S pair called swstat2sw

For example, if the SWSTAT is 37904, the decode can be shown graphically with the command

filterstate $(swstat2sw 37904)

which produces the image shown below.

TITLE: 06/22 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Commissioning work all day after 9, currently locked at 20W in INCREASE_POWER.
LOG:

• 14:00 - Chris S to EX to retrieve tractor and water/1ton truck and then bring it back to the CS. Should be a couple hours of work.
• 15:07 - Richard to LVEA to swap sat amps
• 15:28 - Fil to LVEA to put tester boxes on ITMY M0 and R0
• 15:30 - Vac group tour in LVEA
• 16:00 - Fil, tour out
• 18:02 - Jeff B to Optics Lab
• 18:06 - Fil, Peter K to Optics lab
• 18:14 - Jeff B out
• 18:19 - Fil, Peter out
• 21:44 - Kyle to MY for CP3 fill
• 22:08 - Kyle back

Since the calibration for the rotation stage doesnt seem to last too long, I added a bootstrapping method from rs_power_control.py (that I think Jamie made?)

At the end of the RS move, the bootstrap will move (requested_power^2) / (current power) further. This assumes that it is already close.

We tried it once going from 9.2W to a request of 20W and it went to 20.0W. Very nice.

Conor, Jeff, Jim, Robert, Krishna

- Maximising coherence between the GND STS2 and the BRS is not necessarily the right move.
- Small gains at low frequencies might be made by moving the GND STS2 by some 10s of centimeters.
- Designing sensor-correction filters using offline data provides immediate quantitative performance metrics.

Robert's aLOG of ~1 month ago, 27170, showed 
that coherence between seismometers (and the BRS) changes very rapidly with distance, presumably due to warping of the 
floor or the wind acting like incoherent local sources. It was also apparent the magnitude of the tilt changed by a 
factor of ~5 or more from the edge of the slab to a position far from the walls. 

I wanted to see what the BRS-corrected seismometer performance is like as a function of position. Unfortunately the 
BRS-corrected sensor isn't saved to science frames (H1:HPI-ETMY_STSINF_C_Y_IN1_DQ), and it really should be. On the 
other hand it made me import all the appropriate filters into Matlab to perform offline sensor correction. This allows 
me to evaluate whether we should move the Ground STS2 to a position that is more coherent with the BRS. Attachment 1, 
'GND_spectra.png' shows that the ground motion spectrum is roughly equivalent in the three cases I looked at:
1) 06 May, STS2 huddle
2) 08 May, PEM-STS2 right next to BRS center
3) 13 May, PEM-STS2 at a position with low tilt, on the far side of the BSC, as far from the walls as possible.
These correspond to points 1, 3, and 6 from Robert's post.

1) Huddled STS2s, attachment 2, 'Huddle_BRS_correction.png'
Sensor correction works nearly identically on both devices, using the low-frequency cumulative RMS as a figure of merit. 
Note that I only accumulate RMS below the microseism to better see sensor correction performance.

To get this level of sensor correction, I had to multiply the BRS output by 0.85 for the GND device, and 0.7 for the 
PEM device (in addition to the 'match' filter of 0.85). This means that, even when huddled and coherent, there is ~18% 
less tilt in the PEM device. This is the cause of the better low-f performance, where incoherent BRS noise dominates.

2) PEM next to BRS center, attchment 3, 'BRS_center_BRS_correction.png'
The coherence between the STS2 and the BRS at this position is slightly higher than at the GND position. Perhaps 
this is visible near to 0.1Hz. However, to get good performance, I had to multiply the BRS out by 1.5. This factor is 
solely responsible for the worse low-frequency performance.

3) PEM at low-tilt position, attachment 4, 'low_tilt_BRS_correction.png'
Due to its 5.5m distance from the BRS and GND-STS2, there is almost coherence below 0.1 Hz. The best RMS performance 
comes, unsurprisingly, with no BRS sensor correction. The corrected seismometer wins by a factor ~2 down to 10mHz.

My conclusion from all this is that: Moving the GND-STS2 to increase coherence is not the correct thing to do. 
Moving it ~40cm may surprisingly result in a 18% reduction of noise injection from the BRS with no real loss. 
There may be additional gains to make in this fashion. Moving the BRS to a lower-tilt position, along with the 
GND-STS2 may help, but ideally the BRS will be in a place with higher, but still coherent tilt.

Conclusion #2: while looking at this sensor correction, it became clear that significant gains can be made in the 
BRS sensor correction performance both in global gain and in the high-pass BRS blending filter. I will try to find 
a long, windy period for 'training' this filter.

Conlusion #3: placing STSs in low-tilt positions (like STS_ITMY in the beer garden) is already quite beneficial, at 
the factor 2 or more level. The low-tilt position at ETMY seems to perform as well as the buried seismometer, and for 
some reason much better during the 13 May stretch of data.

Nergis, Peter, Stefan,

Currently, as we transition from 10W to 57W input power, our recycling gain drops from about 34 to 27. It seems like we need to tune the common TCS:

Attached are 3 plots:

Plot 1: DC signals:

PR_GAIN and normalized arm power (both blue)
AS_DC A and B (green and red)
POP_DC A and B (cyan and purple)
REFL_DC A and B (yellow and black)

Note the interesting behaviour of the REFL DC, all while AS_DC is linear in power.

Plot 2:

Arm and power recycling cavity powers, as well as test mass pitch control signals. The test masses have to compensate for radiation pressure, making the pitch control signal proportional to the arm cavity power.

Plot 3: RF signals:

Power and signal recycling cavity sideband buildups.

Jenne, Sheila, Haocun

We took measurements on the hard loops with 40w (actually 37w) laser power today.

The results are plotted with the modeling.

David.M, Jenne.D

Yesterday I set up the sensors for the Newtonian noise L4C array in the beer garden. There are 31 sensors there in total (30 for the array and 1 spare), I arranged them in a huddle beneath the stairs near the STS-2. 20 of them are plugged into the two chassis located in the beer garden which correspond to the first 20 L4C channels. The other chassis is located external to the beer garden next to HAM2, I only connected two sensors to this rack because of a lack of long cables. The channels with these two sensors are L4C channels 26 and 30. I've attached a table which states which serial number L4C is connected to which channel for reference. Also attached are a photo of the current sensor huddle as well as an initial plot of the sensor spectrums calibrated to the STS-2 (thick black line).

The sensors themselves are each only touching the floor (despite what it looks like in the photo), although each wire touches many other wires and don't have any proper strain relief yet. The sensors currently being recorded are the ones more closely huddled together in the photo. The ones which are seperated slightly off to the left are the currently unrecorded L4Cs.

Why is the ETMY motion so different in character than all the other test masses?  All 4 test masses have angular control applied right now, but ETMY looks crazy.  I see it on the oplevs, as well as the L2 OSEMs, so I think it's real motion.

1445 - 1500 hrs. local -> To and from Y-mid 

Opened CP3 exhaust check-valve bypass valve, opened LLCV bypass valve 1/2 turn -> LN2 @ exhaust in 59 seconds -> Restored valves to as found.  

Next manual over-fill to be Friday, June 24th.  

The guardian is currently set up to go to 40W requested (actualkly 37W).. With the interferometer sitting at this power level for ~1h I picked a possible SRC input matrix: Both A36I and B36I have signal, but different offsets. The combination below brings us to a good sideband buildup:

for SRC1_P:
AS_A_RF36_I : -0.5
AS_B_RF36_I : +1.0

for SRC1_Y:
AS_A_RF36_I : -1.5
AS_B_RF36_I : +0.34

Attached are snapshots of the input matrices at 37W

The attached plot shows the arm cavity higher order mode spacing as calculated by Aidan's on-line thermal simulation (TCS_SIM). At 37 W input, the HOM spacing in each arm cavity increases by 60 Hz (X arm) and 75 Hz (Y arm) - relative to the no power-loading case. The ring heater configuration was constant during this time: both cavities had 1 W (total) RH power on the ETM, and no RH power on the ITM. The plot shows that the HOM spacing increases by 30-40 Hz in going from 18 W to 37 W input. If the 15.5 kHz PI modes are stable at 18 W, then we can increase the ETM RH powers to reduce the HOM spacing by 30-40 Hz, and they should then be stable at 37 W. This requires additional RH power of 0.5 W (total, top + bottom).

Couple things to note:

The X-arm simulation output shows a long time constant, but the Y-arm does not. The simulation does include a low pass filter for the surface change due to laser beam self-heating, to simulate this heating time constant. However, this low-pass filter was only engaged for ITMX. I have turned on the filter (surfTF, in FM1) now for all 4 test masses.

The arm cavity power calculation used in the simulation seems to be off. At 37 W input, it is using about 75 kW for both arms, when it should be closer to 150 kW. Something to look into.

Similar to the BSC look here.  Just looking at the higher frequency noise level indicating a problem.  All the HAMs look fine in this regard.  See the attached from diaggui template:

/ligo/svncommon/SeiSVN/seismic/HAM-ISI/H1/Common/CPS_Sensor_Spectra_HAM_ISI.xml

See logs 27648 27622 27549 27479 & 27475 for various looks & reports of the noise seen on BSC CPS channels.

In 27622, it is seen that ITMX Stage 1 has a very noisy V1 channel.  This morning's view has no such issue for any of the stage1 CPSs (this was also reported in 27648.)  The stage2 H3 channel however does now show a slight increase of noise relative to the others.   All other BSC CPSs look good.

We had three PI modes - 15522 of ITMX and 15541 of ETMX and 15542 or ETMY- ring up tonight and were only able to damp successfully ~40 W for a few hours; around 4am local time ITMX and ETMY began ringing up (despite fully driving both ESDs) and broke the lock. 

At 4:42 UTC we originally lost lock from ITMX and ETMX modes ringing up, shown below:. 

 In low power, I drove and damped each test mass to match it with it's peak. Here is a spectrum in low power with peaks labeled for this 15540 Hz mode group.

Tonight, 15521 Hz on ITMX was first damped easily but 15541 on ETMX is tricky since we don't have LV ESD on ETMX until further in the locking process. As Sheila mentioned, we tried switching to LV ESD ETMY so I could turn the LV ESD on for ETMX, but had some troubles and ultimately opted to not damp ETMX tonight but rather try remaining < 50 W. By 2 am we were successfully damping ITMX and (preventatively) ETMY ~40 W. A bit before 4am local, 15521 (ITMX), 15542 (ETMY) began ringing up. I'm having trouble grabbing data during these stretches to track the line amplitudes, so here's a spectrum showing the peak amplitude progression (in UTC times). It looks like ITMX led the ring up. Its not clear to me if ETMX went unstable.

We'll need to play with phases and iWave today to see if that helps strengthen damping.

Damping directions (for now) for 15522 of ITMX, 15541 of ETMX, 15542 of ETMY: 

1. From orange PI button on sitemap, choose ITMX / ETMX / ETMY --> OMC OVERVIEW --> MODE2 / MODE 1 / MODE 4

2. BP and DAMP filters are populated correctly. DAMP filter gain around saturation with sign - / ? / +

Corey, Nergis, Stefan, Sheila, Terra

We were locked above 50 Watts for ~10 minutes.  Our high bandwidth HARD loops along and the ISS 3rd loop are keeping us stable enough to power up, but we still have some things to work out. 

• Corey found that we were saturating a few diodes above 40 Watts.
  • We lowered the whitning gain for all the AS36 WFS by 6dB, (and doubled the gain in the filter modules to compensate), and redid the dark offsets.  
  • We also lowered the whitening gain for LSC-POP_A I and Q, by 6dB, and redid the dark offsets. 
  • For POPX, we turned off one stage of whitneing filters, and reduce the whitening gain by 9 dB, but didn't do dark offsets as this was done in lock. (We need to check these dark offsets in the morning).
• We have been using the reworked ISS 3rd loop, engaging it by hand with a gain of -1 around 15 Watts and leaving it on as we power up, which seems to work well and takes care of the CSOFT instability.  We've been watching the diffracted power as we power up and it seems to stay stable around 15% (4th attached screenshot shows setings)  Durring our last locking attempt the 3rd loop had a prblem, which we have not investigated, around 9:18 UTC.
• We've been using a new plant inversion for CSOFT P, which has lower zeros. (1st screenshot is a comparison of the new (red) control filter vs old (blue), second is a model of what the loop should be like with 25 Watts, but with the gain not scaled correctly).  We expect that the ugf is now around 50 mHz, which is 20 dB higher than the old ugf.    The settings for this loop are now gain -0.045, FM 4,5,6,7,8,9, which is added to guardian, the 3rd attachment shows a measurement of this loop taken at 2 Watts. 
• Stefan has been adjusting the CO2 laser to 0 Watts on ITMY and 0.2 Watts on ITMX.   Stefan added the rotation stage adjust to the ISC_LOCK guardian, but we can move it to the TCS guardian tomorow. 
• Stefan has been using the AS36 -> SRM input matrix, from sunday.  We had been opening the loop to power up past 15 Watts, and adjusting SRM a little bit by hand.  The guardian currently sets the new matrix for SRM and BS, then turns off the SRC1 once we reach 10 Watts.
•  We've added states to the LASER_PWR guardian that go up to 60 Watts, so that we can actually get 50 Watts into the IMC.
• The bounce mode notches are on in the arm ASC loops, which we expect to cause gain peaking.

Our first high power lock broke as we continued to increase the power beyond 50 W and a PI rang up. 

Durring our second attempt Terra damped one PI, (separate alog coming) but we lost lock by increasing the power, ringing up a different PI, and ringing up the bounce modes.

We attempted to switch to ETMY at 20W so Terra could try damping a PI on ETMY, but found that we saturated the low noise driver with the high ASC noise we have right now.  We engaged the Jenne Low passes in CHARD, which stopped the saturations at 20W, but we started to saturate again and lost lock as we tried to increase the power on ETMY.   We will probably need to figure out how to power up on the low noise driver so that we have the option of damping PI on all test masses.

In our last attempt we let the guardian power up to 40 Watts, the power up took about 10 minutes, the only steps we did manually were engaging the ISS 3rd loop and turning off the bounce mode damping.  We lost lock at 40 Watts around 9:18 UTC because of a problem with the ISS 3rd loop.

Richard, Filiberto, Ed, Betsy

Following on from Andy's probing alog 27841 regarding funny looking OSEM spectra, today we looked into a few fishy signals.  Based on spectra he took from before and after the early June power outage which show signal changes, he and Jenne identified the following set of problematic OSEM signals.  I've annotated the list with status as to what we found or fixed today:

ITMY L2 LR - Fixed after power cycling the Satallite Amp box (see alog below)

ITMY R0 RT - Signal looks funny before power outage, old problem, TBC...

MC1 M3 UL - Signal looks funny before power outage, old problem, TBC...

PRM M2 UR - Fixed after power cycling the Satallite Amp box (see alog below)

PR2 M1 T2 - TBC...

PR2 M3 UL - Giant nominal YAW Bias which has been on this SUS for over a year - very little signal on OSEM - mechanical fix when vent

PR2 M3 LL - Giant nominal YAW Bias which has been on this SUS for over a year - very little signal on OSEM - mechanical fix when vent

SR2 M1 LF - Funny comb feature, TBC...

SR2 M1 T1 - Funny comb feature, TBC...

SR2 M1 T3 - Funny comb feature, TBC... 

ETMX L2 LL - 50Hz turn up noise, TBC... Turn up is due to LOCK ACQ PUSHING, turn up not present during nominal SUS damping, no LOCK ACQ, see below plot

ETMX L2 LR - 50Hz turn up noise, TBC... Turn up is due to LOCK ACQ PUSHING, turn up not present during nominal SUS damping, no LOCK ACQ, see below plot

ETMX L2 UR - 50Hz turn up noise, TBC... Turn up is due to LOCK ACQ PUSHING, turn up not present during nominal SUS damping, no LOCK ACQ, see below plot

ETMY L2 UR - 50Hz turn up noise, TBC... Turn up is due to LOCK ACQ PUSHING, turn up not present during nominal SUS damping, no LOCK ACQ, see below plot

IM3 M1 LL - TBC...

We plan to pursue ITMy R0 and SR2 this week at next opportunity.

J. Kissel

After measuring charge today, I've flipped the BIAS sign on both ETMX and ETMY. 

In addition, I've changed the compensation scheme downstream of the BIAS flip such that signs are flipped in each quadrant's ESDOUTF bank GAIN instead of being convoluted in the the DRIVEALIGN matrix elements. This separates the functionality, compensates any angular control as well as longitudinal (though we don't send any angular control to the test mass stage at the moment), and minimizes the possibility for losing the fine-tuned DRIVEALIGN gains. 

Finally, I've modified the ISC guardian in such a way that it monitors the BIAS, and takes care of flipping all of the additional settings necessary to compensate for the bias change in the DOWN state. It had formerly had the DRIVEALIGN gain values hard coded, which was already bad news. Now the ESDOUTF gains (which have not been tuned / balanced as of yet) are flipped between -1 and +1; much easier to maintain. Also, I've added changing the CAL-CS settings to match as well, so the change in calibration will be taken care of as well.

I would post the latest results from today's change measurements, but I've been stuck trying to get 1 second of EPICS data from NDS2 for about 4 hours now.

Details
------------------
Before Flipping (to preserve my sanity, and minimize minus signs):
- Moved DRIVEALIGN negative signs into the ESDOUTF bank:
          ETMX              ETMY
      OLD       NEW       OLD     NEW
L2L   -1          +1       -30     30
L2P   -0.021  +0.021         0      0
L2Y   -0.007  +0.007         0      0
and the ESDOUTF GAINs had all been positive unity, were flipped to negative unity so as to not yet actually flip the compensating sign.

Then, I changed the ETMX bias sign:
- Changed H1:SUS-ETMX_L3_LOCK_INBIAS from +9.5 to -9.5 [V_DAC]
- (unlike before) The ISC_LOCK guardian (now) takes care of checking the sign of H1:SUS-ETMX_L3_LOCK_INBIAS, and adjusting the sign of 
   - H1:SUS-ETMX_L3_ESDOUTF_[UL,LL,UR,LR]_GAIN from -1.000 to +1.000  (fixes the ESD stage longitudinal loop gain to match the bias sign change)
   - H1:SUS-ETMX_L3_ESDOUTF_LIN_FORCE_COEFF from +124518.4 to -124518.4 (fixes the linearization force coefficient to match the new bias sign)
   accordingly.
- (like before) Accepted the new values in *all* SDF snap files, 
   /opt/rtcds/userapps/release/sus/h1/burtfiles/
   - h1susetmx_down.snap
   - h1susetmx_observe.snap
   - h1susetmx_safe.snap
  by loading in each table and accepting the values I've changed, followed by a commit to the userapps repo. 
- (like before) There's no need to make any further changes in the CAL-CS epics settings, because ETMX is not used in any capacity for calibration.

Followed by the ETMY bias sign:
- Changed H1:SUS-ETMY_L3_LOCK_INBIAS from -9.5 to +9.5 [V_DAC] 
- (unlike before) The ISC_LOCK guardian (now) takes care of checking the sign of H1:SUS-ETMX_L3_LOCK_INBIAS, and adjusting the sign of 
   - H1:SUS-ETMY_L3_ESDOUTF_[UL,LL,UR,LR]_GAIN from -1.000 to +1.000  (fixes the ESD stage longitudinal loop gain to match the bias sign change)
   - H1:SUS-ETMY_L3_ESDOUTF_LIN_FORCE_COEFF from -124518.4 to +124518.4 (fixes the linearization force coefficient to match the new bias sign)
   accordingly.
- (like before) Accepted the new values in *all* SDF snap files, 
   /opt/rtcds/userapps/release/sus/h1/burtfiles/
   - h1susetmy_down.snap
   - h1susetmy_observe.snap
   - h1susetmy_safe.snap
  by loading in each table and accepting the values I've changed, followed by a commit to the userapps repo. 
 (This also involved *creating* a copy of the down.snap in the userapps repo, and changing the file in the 
  target/h1susetmy/h1susetmyepics/burt/ directory to a soft link, as was already the case for the OBSERVE.snap and safe.snap) 

[Making sure Calibration is unaffected]
- (like before) We must make sure that the CAL-CS replica of the ETMY actuation matches the SUS ETMY digital signal chain. However, because the CAL-CS model had not been restarted, and its SDF system had been kept up-to-date it did not lose the appropriate settings, so the settings for
   - H1:CAL-CS_DARM_FE_ETMY_L3_DRIVEALIGN_L2L_GAIN to +30.0
   - H1:CAL-CS_DARM_FE_ETMY_L3_ESDOUTF_UL_GAIN to +1
   - H1:CAL-CS_DARM_ANALOG_ETMY_L3_GAIN stays +1.0
- (unlike before) Again, to be consistent, I've changed the sign of the linearization force coefficient,
   - H1:CAL-CS_DARM_FE_ETMY_L3_ESDOUTF_LIN_FORCE_COEFF from +124518.4 to -124518.4
- (unlike before) Accepted the new values in *all* SDF snap files, 
   /opt/rtcds/userapps/release/sus/h1/burtfiles/
   - h1susetmx_observe.snap
   - h1susetmx_safe.snap
  by loading in each table and accepting the values I've changed, followed by a commit to the userapps repo. 
 (Since there is no change between the "down" and "observation" state, a "down" .snap doesn't and need not exist.)