Related logs, 31303, 31371, 31403

Jeff, Kiwamu,

We have copied the actuator function plotting script from LLO and adopted for LHO. The code seems to run fine. It resides at

/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER10/H1/Scripts/FullIFOActuatorTFs/actuatorCoefficients.m

The attached are the resulting pdf showing the three sets of measurements that Jeff took in the past few days. I did not do fittin yet. Below are several notes

• We have incorported the correction factors for the actuator functions (31369).
  • They are directly  writen in the interferometer dependent paramter file at /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER10/H1/params/H1params.conf
• We are reminded that there is significant discrepancy between the measured and model in UIM at high frequencies (24197)
• TST and PUM seem fine both in magnitude and phase.
• We switched the suspension tag model
  • from quadmodelproduction-rev7995_ssmake4pv2eMB5f_fiber-rev3601_h1etmy-rev7915_released-2016-06-15.mat
  • to quadmodelproduction-rev7995_ssmake4pv2eMB5f_fiber-rev3601_h1etmy-rev7915_released-2016-03-01.mat
• However, in doing so, Jeff found what appears to be a bug in the suspension model which can impact on low frequency suspension resonances around 1 Hz. He is working on this at the moment.

Operator task request

Now that we're at lower power, we'd like to double check what PI's actually need to be damped. We still have damping settings set up on all the modes that rang up while at 50 W, but it looks like most of them are no longer actually needed. I trended the past several days of locks and only Modes 26, 27, 28, (and maybe 3) rang up enough to engage their damping loops. We'd like to turn off the damping for the other modes and watch and make sure that they really don't need it. 

To check:

1. Once you've passed DC READOUT, guardian has turned on all the usual PI gains. Write the gains down (or I've put them in the comment below) so that you can quickly put these values back in if needed.

2. Set the gains of Modes  1 - 25   (edited to now include keeping Mode 3 on) 1, 2 , 4 - 25 to zero. Do Not zero Modes 3, 26, 27, 28 as we are definitely actively damping those. Just do this directly on the main PI medm screen under the 'gain' column. 

3. Watch the modes throughout the lock. The StripTool will still give the same ring up info as always and the DTT screen is a good way to keep an eye on if something is growing. 

4. If a mode starts to clearly ring up, put back in the original noted gain value and damp as usual. Make a note in the log. 

I've chosen not to just auto set the gains to zero already in case the operator is busy for now. Let's get a few longer locks that are more closely watched and confirm we really dont need damping on. 

Fil, Jim

We replaced the ADC0 card, ribbon cable, and interface card in the h1oaf0 I/O chassis after the last failure of the ADC/DAC.  We also moved the h1oaf0 timing fiber and SFP on the timing fanout from port 1 to port 13.  (This causes an error status on the Beckhoff timing MEDM screen, which shouldn't cause problems.)

After startup, we noted that the timing status on the IOP model flashed several times, this can be observed in a raw data plot of the H1:FEC-23_STATE_WORD channel for 10 minutes ending at GPS 1162946454.  See attached plot.  We have not seen this behaviour previously.

Present status:

h1oaf0 I/O chassis replaced with x1psl0 DTS I/O chassis, but using all ADC/DAC/BIO and interface cards with the exception of ADC0 and interface card, which were replaced with a set from the x1psl0 I/O chassis.

State of H1: locking well, OAF issues

Activities:

• all locking attempts made it to NLN
• all NLN attempts included waiting in Noise Tunings for at least 15 minutes
• OAF has died multiple time
• One lock was saved when OAF came back and I was able to damp PI mode (no modes were ringing up when OAF died)
• I suggested we might save more locks if the PI mode damping was down as short a time as possible
• Jim reorganized the OAF restart script to kill the PI process last, and restart the PI process first, to minimize the down time
• I've taken H1 out of lock twice today to accomodate invasive OAF work
• one lock loss was during the earthquake at approx. 22:00UTC

Currently:

• fire department is at MX to look at their fire pannel - it didn't respond to the daily query
• Fil and Jim are here and trying another change on OAF
• OAF is currently down

In an effort to get the h1oaf0 to run reliably, Fil and I replaced the I/O chassis with the x1psl0 chassis from the DAQ test stand, which has been the most reliable.  

We used the same ADC/DAC/BIO cards as were in the original h1oaf0 I/O chassis.

The ADC/DAC failed within 5 minutes.

Next we will try swapping ADC0 and it's interface card.

2:44pm local

Filled CP3 from control room today! Took 26 min. Set LLCV to 50% open (from 20%) and monitored the two thermocouple readings on MEDM. 

Since it took so long I increased nominal to 21%. 

CP3 Dewar is scheduled for fill on Tuesday. 

T240:

Averaging Mass Centering channels for 10 [sec] ...
2016-11-11 12:46:08.555164

There are 4 T240 proof masses out of range ( > 0.3 [V] )!
ETMY T240 3 DOF Z/W = 0.328 [V]
ITMX T240 1 DOF X/U = -0.47 [V]
ITMX T240 3 DOF X/U = -0.46 [V]
ITMY T240 3 DOF Z/W = -0.743 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = 0.093 [V]
ETMX T240 1 DOF Y/V = 0.075 [V]
ETMX T240 1 DOF Z/W = 0.105 [V]
ETMX T240 2 DOF X/U = 0.088 [V]
ETMX T240 2 DOF Y/V = 0.054 [V]
ETMX T240 2 DOF Z/W = 0.111 [V]
ETMX T240 3 DOF X/U = 0.051 [V]
ETMX T240 3 DOF Y/V = 0.022 [V]
ETMX T240 3 DOF Z/W = 0.061 [V]
ETMY T240 1 DOF X/U = -0.03 [V]
ETMY T240 1 DOF Y/V = -0.014 [V]
ETMY T240 1 DOF Z/W = -0.2 [V]
ETMY T240 2 DOF X/U = 0.204 [V]
ETMY T240 2 DOF Y/V = -0.201 [V]
ETMY T240 2 DOF Z/W = 0.011 [V]
ETMY T240 3 DOF X/U = -0.221 [V]
ETMY T240 3 DOF Y/V = -0.04 [V]
ITMX T240 1 DOF Y/V = 0.237 [V]
ITMX T240 1 DOF Z/W = 0.179 [V]
ITMX T240 2 DOF X/U = 0.245 [V]
ITMX T240 2 DOF Y/V = 0.204 [V]
ITMX T240 2 DOF Z/W = 0.232 [V]
ITMX T240 3 DOF Y/V = 0.177 [V]
ITMX T240 3 DOF Z/W = 0.212 [V]
ITMY T240 1 DOF X/U = 0.135 [V]
ITMY T240 1 DOF Y/V = 0.036 [V]
ITMY T240 1 DOF Z/W = 0.036 [V]
ITMY T240 2 DOF X/U = 0.071 [V]
ITMY T240 2 DOF Y/V = 0.172 [V]
ITMY T240 2 DOF Z/W = 0.163 [V]
ITMY T240 3 DOF X/U = -0.195 [V]
ITMY T240 3 DOF Y/V = 0.171 [V]
BS T240 1 DOF X/U = 0.028 [V]
BS T240 1 DOF Y/V = 0.053 [V]
BS T240 1 DOF Z/W = 0.202 [V]
BS T240 2 DOF X/U = 0.135 [V]
BS T240 2 DOF Y/V = 0.184 [V]
BS T240 2 DOF Z/W = 0.174 [V]
BS T240 3 DOF X/U = 0.031 [V]
BS T240 3 DOF Y/V = 0.082 [V]
BS T240 3 DOF Z/W = -0.014 [V]

Assessment complete.

---------

STS:

Averaging Mass Centering channels for 10 [sec] ...

2016-11-11 12:47:53.351902
All STSs prrof masses that within healthy range (< 2.0 [V]). Great!

Here's a list of how they're doing just in case you care:
STS A DOF X/U = -0.537 [V]
STS A DOF Y/V = -0.053 [V]
STS A DOF Z/W = -0.503 [V]
STS B DOF X/U = 0.098 [V]
STS B DOF Y/V = -1.23 [V]
STS B DOF Z/W = -0.064 [V]
STS C DOF X/U = -0.0 [V]
STS C DOF Y/V = -0.0 [V]
STS C DOF Z/W = -0.0 [V]
STS EX DOF X/U = -0.21 [V]
STS EX DOF Y/V = 0.553 [V]
STS EX DOF Z/W = 0.086 [V]
STS EY DOF X/U = 0.027 [V]
STS EY DOF Y/V = 0.315 [V]
STS EY DOF Z/W = 0.298 [V]

Assessment complete.

WP 6314

Fil, Jim

In an attempt to stabilize the h1oaf0 ADC/DAC issue, Fil and Jim replaced the DC power supply in the h1oaf0 I/O chassis, and reseated all ADC, DAC, BIO, and interface cards in the chassis.  Start time 9:31, power up of I/O chassis at 9:55. The first attempt at starting the models resulted in ADC and DAC errors like we've seen, and the IRIG-B time going negative.  The models were killed, then restarted manually to watch for issues.  The models started OK on the second attempt.  

Note that the TCS AI chassis was powered down during this work, and was powered back up once the h1tcscs model was running.  We avoided tripping the TCS laser chillers.

• H1 schedule - Mike, Keita
  • for LHO ER10 starts Monday Nov. 14th at 8AM PT, 16:00UTC
  • decision about O2A start in process
  • potential start dates are currently Nov. 28th-Dec 5th
  • 1PM Engineering Run meeting every day, also weekends
  • Rapid Response Team procedures are being looked at / updated
• H1 - Cheryl
  • H1 is relocking well
  • OAF script to restart worked well last night and over night
  • PI modes are more troublesome than in O1, being looked at, Terra is here
  • Sheila says pause in Noise Tunings to let REFL WFS center and settle before continuing on
• CDS - Jim, OAF instability is new, being looked at
• VAC - Chandra, cathode gauges 7 and 8 drifting up, investigation in progress
  • might need to run a pump for 24 hours to evaluate
  • might need to prep this today for work next Tuesday
• Facilities - Bubba, actively looking for water leak
• Tour on Saturday

TITLE: 11/11 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Locking has been very straight forward tonight, two lockloss from unknown reasons, but other than that I have been in observing. Sheila suggested that I wait at NOISE_TUNINGS for 10-15 before moving on, you can run a2l while waiting. ITMY is still swinging, but not as much as I saw last weekend.
LOG:

• 10:48 Observing
• 13:54 Lockloss. Not sure of the cause, everything viually looked good. Went out to heat up food while ITMY settled, came back and the OAF crashed. Restarted using Dave's script with no issues.
• 14:49 Observing
• 15:30 Lockloss
• 16:15 Observing

The other night I ran Stefan's spot move script, which tracks the alignment of all optics based on the combination of ASC control signals and sliders.  Tnight I tried to imitate that move of the interferometer alignemt by moving IMs, by moving IM1 I was able to see a small improvement in recycling gain, (from about 30.7 to 30.8) but no improvement in the 260Hz jitter peak.  I started trying to move IM2, but moved too quickly and broke the lock.

When I moved the IMC DOF1P offset to 120 counts this is how the other optics moved:

 PIT: SUS-IM1_M1_ 0.0

SHIFT SUMMARY: Commissioning all evening. Oaf computer crashed at least twice this evening. Dave created a script to kill and restart the model that can be un anywhere from ops computer. Just do ./restart_h1oaf0_models.bsh. Be mindful that when the models are being restarted PI won't be damped. So be ready to damp PI after every oaf restart because some of the modes will come back high every time.
 

We measured the pitch refl sensing matrix for PR3, IM4 and CHARD again, and adjusted the input matrix for PR3 a little.  This helped to keep our sideband build ups much mroe stable after we transitioned to it and the interferometer seems stable like this so far.  

The old matrix for PR3 was 0.08 REFL B 9I -0.08 REFL B45 I.  THe new matrix is 0.1 REFL B 9 I -0.0768 REFL B 45I 

It seems like we are on the edge of the recycling gains where the REFL WFS have a decent signal for PR3, so that the matrix we need to use changes with the recycing gain.  We've had a couple more locklosses when switching to the new matrix, but it seems to work if we wait ~15 minutes after powering up to try the switch, but it works if we .  I"ve moved the switch from POPX WFS to a new state called PR3_REFL_WFS, so that operators can easily skip it and beam diverters if they need to.  For now I will leave it in the guardian with the advice to operators to wait a while before trying it.  If you are having locklosses quickly after this state, then skip it and skip the beam divereters as well. 

To speed up the restart we have created an SSH key-pair between the operator station and h1oaf0, and written a script to kill and restart the models in the correct sequence.

1. log into operator0 as user ops (password in shared secrets)

    ssh ops@operator0

2. run script

    ./restart_h1oaf0_models.bsh

When the models are running again, at the bottom of the CDS Overview MEDM press the buttons "!Press All Diag Reset Buttons" and "!DAQ Clear Accumulated CRC" to green up the board.

If problem persists, we will do more hardware investigations tomorrow

We measured the REFL signals at 2W with the ISS second loop engaged and off (REF traces). The RF signals measure residual RF modulation (RAM), whereas LF measured the RIN. The REFL PD sees about a factor of 10 more  light than the ISS second loop PDs when unlocked.

There seems to be a wide scatter peak at ~16.5 Hz. The 9 MHz demodulated signal shows some strange smooth excess noise below 100 Hz, whereas the 45 MHz seems to see more of the jitter peaks. The 9 MHz signal has been scaled to give the same digital and electronics gain as the 45 MHz. The transimpedance gains are very similar to start with.