WP 6314

Fil, Jim

In an attempt to stabilize the h1oaf0 ADC/DAC issue, Fil and Jim replaced the DC power supply in the h1oaf0 I/O chassis, and reseated all ADC, DAC, BIO, and interface cards in the chassis.  Start time 9:31, power up of I/O chassis at 9:55. The first attempt at starting the models resulted in ADC and DAC errors like we've seen, and the IRIG-B time going negative.  The models were killed, then restarted manually to watch for issues.  The models started OK on the second attempt.  

Note that the TCS AI chassis was powered down during this work, and was powered back up once the h1tcscs model was running.  We avoided tripping the TCS laser chillers.

• H1 schedule - Mike, Keita
  • for LHO ER10 starts Monday Nov. 14th at 8AM PT, 16:00UTC
  • decision about O2A start in process
  • potential start dates are currently Nov. 28th-Dec 5th
  • 1PM Engineering Run meeting every day, also weekends
  • Rapid Response Team procedures are being looked at / updated
• H1 - Cheryl
  • H1 is relocking well
  • OAF script to restart worked well last night and over night
  • PI modes are more troublesome than in O1, being looked at, Terra is here
  • Sheila says pause in Noise Tunings to let REFL WFS center and settle before continuing on
• CDS - Jim, OAF instability is new, being looked at
• VAC - Chandra, cathode gauges 7 and 8 drifting up, investigation in progress
  • might need to run a pump for 24 hours to evaluate
  • might need to prep this today for work next Tuesday
• Facilities - Bubba, actively looking for water leak
• Tour on Saturday

TITLE: 11/11 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Locking has been very straight forward tonight, two lockloss from unknown reasons, but other than that I have been in observing. Sheila suggested that I wait at NOISE_TUNINGS for 10-15 before moving on, you can run a2l while waiting. ITMY is still swinging, but not as much as I saw last weekend.
LOG:

• 10:48 Observing
• 13:54 Lockloss. Not sure of the cause, everything viually looked good. Went out to heat up food while ITMY settled, came back and the OAF crashed. Restarted using Dave's script with no issues.
• 14:49 Observing
• 15:30 Lockloss
• 16:15 Observing

The other night I ran Stefan's spot move script, which tracks the alignment of all optics based on the combination of ASC control signals and sliders.  Tnight I tried to imitate that move of the interferometer alignemt by moving IMs, by moving IM1 I was able to see a small improvement in recycling gain, (from about 30.7 to 30.8) but no improvement in the 260Hz jitter peak.  I started trying to move IM2, but moved too quickly and broke the lock.

When I moved the IMC DOF1P offset to 120 counts this is how the other optics moved:

 PIT: SUS-IM1_M1_ 0.0

SHIFT SUMMARY: Commissioning all evening. Oaf computer crashed at least twice this evening. Dave created a script to kill and restart the model that can be un anywhere from ops computer. Just do ./restart_h1oaf0_models.bsh. Be mindful that when the models are being restarted PI won't be damped. So be ready to damp PI after every oaf restart because some of the modes will come back high every time.
 

We measured the pitch refl sensing matrix for PR3, IM4 and CHARD again, and adjusted the input matrix for PR3 a little.  This helped to keep our sideband build ups much mroe stable after we transitioned to it and the interferometer seems stable like this so far.  

The old matrix for PR3 was 0.08 REFL B 9I -0.08 REFL B45 I.  THe new matrix is 0.1 REFL B 9 I -0.0768 REFL B 45I 

It seems like we are on the edge of the recycling gains where the REFL WFS have a decent signal for PR3, so that the matrix we need to use changes with the recycing gain.  We've had a couple more locklosses when switching to the new matrix, but it seems to work if we wait ~15 minutes after powering up to try the switch, but it works if we .  I"ve moved the switch from POPX WFS to a new state called PR3_REFL_WFS, so that operators can easily skip it and beam diverters if they need to.  For now I will leave it in the guardian with the advice to operators to wait a while before trying it.  If you are having locklosses quickly after this state, then skip it and skip the beam divereters as well. 

J. Kissel, H. Radkins

I'm continuing to investigate why ITMY's pitch mode (at 0.55 Hz) continues to ring much longer than the other QUAD's upon lockloss (see and example attached as 2016-11-08_2151UTC_ITMY_Oplev_Bad_TimeSeries.png). 

Today, I grabbed the open loop gain transfer functions of the M0 DAMP L, the M0 DAMP P, and the L2 OLDAMP P local damping loops on both H1 SUS ETMY and H1 SUS ITMY, to show that the identical settings are producing identical results for QUADs that are essentially physically identical. 

Indeed, the open loop gain transfer functions of both suspensions look the same.

We don't have too many lock loss data points yet since I made the change, but I did discover that the digital output limiter of all QUAD's L2 OLDAMP P bank were different (one as low as 5000, and one as high as 500000, and ITMY was in the middle at 10000). So, I've made all the limits the same, at 40000, in hopes that the problem is with the square-wave that ITMY gets sent during a lock loss. This change is NOT accepted in the SDF, because I'm unsure if it matters yet.

Still unclear what is going on here...

Hugh's taking a look at these same optical lever signals after every lock loss for the past month. Preliminary results show that this behavior is intermittent, but has definitely gotten worse in the past week or two. We hope he can trace this back to some event like a maintenance day to better try to address the problem.

To speed up the restart we have created an SSH key-pair between the operator station and h1oaf0, and written a script to kill and restart the models in the correct sequence.

1. log into operator0 as user ops (password in shared secrets)

    ssh ops@operator0

2. run script

    ./restart_h1oaf0_models.bsh

When the models are running again, at the bottom of the CDS Overview MEDM press the buttons "!Press All Diag Reset Buttons" and "!DAQ Clear Accumulated CRC" to green up the board.

If problem persists, we will do more hardware investigations tomorrow

J. Kissel

Though it's been a struggle between all of the H1OAF computer crashing, I've managed to squeak out one more round of calibration measurements today. The data's location is below; analysis to come.

The data lives here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER10/H1/Measurements/
(1a) DARMOLGTFs/2016-11-10_H1_DARM_OLGTF_4to1200Hz_fasttemplate.xml
(1b) PCAL/2016-11-10_H1_PCAL2DARMTF_4to1200Hz_fasttemplate.xml

(2)
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L1_iEXC2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L1_PCAL2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L2_iEXC2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L2_PCAL2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L3_iEXC2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L3_PCAL2DARM.xml

We measured the REFL signals at 2W with the ISS second loop engaged and off (REF traces). The RF signals measure residual RF modulation (RAM), whereas LF measured the RIN. The REFL PD sees about a factor of 10 more  light than the ISS second loop PDs when unlocked.

There seems to be a wide scatter peak at ~16.5 Hz. The 9 MHz demodulated signal shows some strange smooth excess noise below 100 Hz, whereas the 45 MHz seems to see more of the jitter peaks. The 9 MHz signal has been scaled to give the same digital and electronics gain as the 45 MHz. The transimpedance gains are very similar to start with.

State of H1: locking and going to NLN

Assistance: Betsy, Hugh, Richard, Jenne, Sheila, JeffK, Jason, Tara

Activities:  all times UTC

• OAF died right before my shift - being worked on when I arrived
• H1 lock loss
• massive confusion about what happens when OAF dies and TCS goes with it
• final observation about how to restart TCS is to
  • leave it completely alone
  • don't touch the TCS guardian
  • don't touch any settings
  • restart the chillers and it will come back
  • it would also be helpful to consider a rework of the medm screen to improve the operator's ability to diagnose issues
• H1 relocked and back to NLN
• OAF dies, Dave and Jim try to bring it back before TCS being off killed the lock, and we ran out of time (new cable, not sure if it helped)
• while H1 was still locked and OAF came back, no PI damping, since the safe.snap gets loaded
• I was working on PI mode 28 as OAF died,  and it was increasing, since I hadn't yet found the new good phase
• PI mode 28 was 0.5 when OAF died, and when OAF came back 16 minutes later, it grew to 1.6 before lock loss
• PI mode 27 was not rung up when OAF died, but when OAF came back it was at 500, and rose to 2500 at lock loss
• when OAF came back, the PI damping was restored to SAFE, so off, and I was not able to locate a file to restore the damping before lock loss
• many parties used the H1 down down time for measurements
  • Jenne, JeffK, Sheila, Daniel, Kiwamu
  • all measurements are done
• H1 is relocked and past DRMI_LOCKED, and heading to NLN

Site Activities:

• Karen to MY, and back
• Chandra to MY, and back
• Fred to LVEA - tour, visitor is the first LIGO engineer and family
• Sheila to LVEA - PSL racks - IMC measurement

Another event at 14:02 PST , proc status file for IOP model posted below.

If this is a timing error (and at this point we are not certain of this), our first change was to replace the fiber optics cable between the IO Chassis timing slave and the timing fanout chassis. A new cable was installed before the system was restarted.

Note, the existing fiber was found to be not cable tied to the bundle of fibers at the timing fan-out card, rather was floating by itself.

Sequence was:

• kill all models in correct sequence
• take h1oaf0 out of the dolphin fabric
• power down h1loaf0 computer
• power down IO Chassis
• replace timing fiber
• power up IO Chassis *
• power up h1oaf0 computer
• wait for irig-b to come back to positive values (approx 3 mins)

* we did not do the TCS-AI power sequence this time and unfortunately this may have tripped the TCS chillers.

controls@h1oaf0 ~ 130$ cat /proc/h1iopoaf0/status
startGpsTime=1162838354
uptime=12257
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162838680
adcHoldTime=16
adcHoldTimeEverMax=91
adcHoldTimeEverMaxWhen=1162850544
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=3
cycle=49476
gps=1162850611
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,5
cpuTimeMaxCycle(cur,past sec)=21,21
cycleHist: 3=65183@17 4=350@65535 5=3@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=91 Current=14
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0

press DIAG_RESET

controls@h1oaf0 ~ 0$ cat /proc/h1iopoaf0/status
startGpsTime=1162838354
uptime=12333
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162838680
adcHoldTime=14
adcHoldTimeEverMax=91
adcHoldTimeEverMaxWhen=1162850544
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=4
cycle=43808
gps=1162850687
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,6
cpuTimeMaxCycle(cur,past sec)=21,1
cycleHist: 3=65201@17 4=333@65535 5=1@0 6=1@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=17 Current=15
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0

 

This is a continuity of 31381.

For a bookkeeping purpose, I have extended the spot position thing to PRM and MC mirrors. Here is a summary. They all seem fine as expected.

The measurement was done with the interferometer fully locked at 25 W in nominal low noise. The measurement precision should be something like 10% or so.