SHIFT SUMMARY: Commissioning all evening. Oaf computer crashed at least twice this evening. Dave created a script to kill and restart the model that can be un anywhere from ops computer. Just do ./restart_h1oaf0_models.bsh. Be mindful that when the models are being restarted PI won't be damped. So be ready to damp PI after every oaf restart because some of the modes will come back high every time.
 

We measured the pitch refl sensing matrix for PR3, IM4 and CHARD again, and adjusted the input matrix for PR3 a little.  This helped to keep our sideband build ups much mroe stable after we transitioned to it and the interferometer seems stable like this so far.  

The old matrix for PR3 was 0.08 REFL B 9I -0.08 REFL B45 I.  THe new matrix is 0.1 REFL B 9 I -0.0768 REFL B 45I 

It seems like we are on the edge of the recycling gains where the REFL WFS have a decent signal for PR3, so that the matrix we need to use changes with the recycing gain.  We've had a couple more locklosses when switching to the new matrix, but it seems to work if we wait ~15 minutes after powering up to try the switch, but it works if we .  I"ve moved the switch from POPX WFS to a new state called PR3_REFL_WFS, so that operators can easily skip it and beam diverters if they need to.  For now I will leave it in the guardian with the advice to operators to wait a while before trying it.  If you are having locklosses quickly after this state, then skip it and skip the beam divereters as well. 

J. Kissel, H. Radkins

I'm continuing to investigate why ITMY's pitch mode (at 0.55 Hz) continues to ring much longer than the other QUAD's upon lockloss (see and example attached as 2016-11-08_2151UTC_ITMY_Oplev_Bad_TimeSeries.png). 

Today, I grabbed the open loop gain transfer functions of the M0 DAMP L, the M0 DAMP P, and the L2 OLDAMP P local damping loops on both H1 SUS ETMY and H1 SUS ITMY, to show that the identical settings are producing identical results for QUADs that are essentially physically identical. 

Indeed, the open loop gain transfer functions of both suspensions look the same.

We don't have too many lock loss data points yet since I made the change, but I did discover that the digital output limiter of all QUAD's L2 OLDAMP P bank were different (one as low as 5000, and one as high as 500000, and ITMY was in the middle at 10000). So, I've made all the limits the same, at 40000, in hopes that the problem is with the square-wave that ITMY gets sent during a lock loss. This change is NOT accepted in the SDF, because I'm unsure if it matters yet.

Still unclear what is going on here...

Hugh's taking a look at these same optical lever signals after every lock loss for the past month. Preliminary results show that this behavior is intermittent, but has definitely gotten worse in the past week or two. We hope he can trace this back to some event like a maintenance day to better try to address the problem.

To speed up the restart we have created an SSH key-pair between the operator station and h1oaf0, and written a script to kill and restart the models in the correct sequence.

1. log into operator0 as user ops (password in shared secrets)

    ssh ops@operator0

2. run script

    ./restart_h1oaf0_models.bsh

When the models are running again, at the bottom of the CDS Overview MEDM press the buttons "!Press All Diag Reset Buttons" and "!DAQ Clear Accumulated CRC" to green up the board.

If problem persists, we will do more hardware investigations tomorrow

J. Kissel

Though it's been a struggle between all of the H1OAF computer crashing, I've managed to squeak out one more round of calibration measurements today. The data's location is below; analysis to come.

The data lives here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER10/H1/Measurements/
(1a) DARMOLGTFs/2016-11-10_H1_DARM_OLGTF_4to1200Hz_fasttemplate.xml
(1b) PCAL/2016-11-10_H1_PCAL2DARMTF_4to1200Hz_fasttemplate.xml

(2)
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L1_iEXC2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L1_PCAL2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L2_iEXC2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L2_PCAL2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L3_iEXC2DARM.xml
FullIFOActuatorTFs/2016-11-10/2016-11-10_H1SUSETMY_L3_PCAL2DARM.xml

We measured the REFL signals at 2W with the ISS second loop engaged and off (REF traces). The RF signals measure residual RF modulation (RAM), whereas LF measured the RIN. The REFL PD sees about a factor of 10 more  light than the ISS second loop PDs when unlocked.

There seems to be a wide scatter peak at ~16.5 Hz. The 9 MHz demodulated signal shows some strange smooth excess noise below 100 Hz, whereas the 45 MHz seems to see more of the jitter peaks. The 9 MHz signal has been scaled to give the same digital and electronics gain as the 45 MHz. The transimpedance gains are very similar to start with.

State of H1: locking and going to NLN

Assistance: Betsy, Hugh, Richard, Jenne, Sheila, JeffK, Jason, Tara

Activities:  all times UTC

• OAF died right before my shift - being worked on when I arrived
• H1 lock loss
• massive confusion about what happens when OAF dies and TCS goes with it
• final observation about how to restart TCS is to
  • leave it completely alone
  • don't touch the TCS guardian
  • don't touch any settings
  • restart the chillers and it will come back
  • it would also be helpful to consider a rework of the medm screen to improve the operator's ability to diagnose issues
• H1 relocked and back to NLN
• OAF dies, Dave and Jim try to bring it back before TCS being off killed the lock, and we ran out of time (new cable, not sure if it helped)
• while H1 was still locked and OAF came back, no PI damping, since the safe.snap gets loaded
• I was working on PI mode 28 as OAF died,  and it was increasing, since I hadn't yet found the new good phase
• PI mode 28 was 0.5 when OAF died, and when OAF came back 16 minutes later, it grew to 1.6 before lock loss
• PI mode 27 was not rung up when OAF died, but when OAF came back it was at 500, and rose to 2500 at lock loss
• when OAF came back, the PI damping was restored to SAFE, so off, and I was not able to locate a file to restore the damping before lock loss
• many parties used the H1 down down time for measurements
  • Jenne, JeffK, Sheila, Daniel, Kiwamu
  • all measurements are done
• H1 is relocked and past DRMI_LOCKED, and heading to NLN

Site Activities:

• Karen to MY, and back
• Chandra to MY, and back
• Fred to LVEA - tour, visitor is the first LIGO engineer and family
• Sheila to LVEA - PSL racks - IMC measurement

Another event at 14:02 PST , proc status file for IOP model posted below.

If this is a timing error (and at this point we are not certain of this), our first change was to replace the fiber optics cable between the IO Chassis timing slave and the timing fanout chassis. A new cable was installed before the system was restarted.

Note, the existing fiber was found to be not cable tied to the bundle of fibers at the timing fan-out card, rather was floating by itself.

Sequence was:

• kill all models in correct sequence
• take h1oaf0 out of the dolphin fabric
• power down h1loaf0 computer
• power down IO Chassis
• replace timing fiber
• power up IO Chassis *
• power up h1oaf0 computer
• wait for irig-b to come back to positive values (approx 3 mins)

* we did not do the TCS-AI power sequence this time and unfortunately this may have tripped the TCS chillers.

controls@h1oaf0 ~ 130$ cat /proc/h1iopoaf0/status
startGpsTime=1162838354
uptime=12257
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162838680
adcHoldTime=16
adcHoldTimeEverMax=91
adcHoldTimeEverMaxWhen=1162850544
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=3
cycle=49476
gps=1162850611
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,5
cpuTimeMaxCycle(cur,past sec)=21,21
cycleHist: 3=65183@17 4=350@65535 5=3@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=91 Current=14
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0

press DIAG_RESET

controls@h1oaf0 ~ 0$ cat /proc/h1iopoaf0/status
startGpsTime=1162838354
uptime=12333
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162838680
adcHoldTime=14
adcHoldTimeEverMax=91
adcHoldTimeEverMaxWhen=1162850544
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=4
cycle=43808
gps=1162850687
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,6
cpuTimeMaxCycle(cur,past sec)=21,1
cycleHist: 3=65201@17 4=333@65535 5=1@0 6=1@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=17 Current=15
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0

 

Here's the report of a BruCo scan for last night's lock

https://ldas-jobs.ligo.caltech.edu/~gabriele.vajente/bruco_lho_1162821617/

Some interesting results:

• in the 10-20 Hz region there is coherence with
  • WFS signals (ASC-REFL_A_RF9_I_PIT, fig. 1),
  • ASC signals (ASC-CHARD_P_OUT, fig. 3, ASC-MICH_Y, fig. 5, ASC-CHARD_Y_OUT, fig. 6, some CSOFT too, fig. 7)
  • but more interesting with HPI signals (for example HPI-HAM1_SCSUM_IPS_IN_Z, but others too, fig. 2)
  • and with RM signals (SUS-RM2_M1_OSEMINF_UR_OUT fig. 4)
• at 70 Hz PEM-EX_MAG_EBAY_SUSRACK_Z
• not much between 40 and 120 Hz
• usual jitter peaks above 100 Hz (fig. 8 for an example, lot of coherence with IMC WFS too), coherence well visible in the IMC MCL signals ad ISS signals too

A question for commissioners, should I exclude channels like SUS-ETMY_L2_FASTIMON_* ??

at 10:30 PST the h1iopoaf0 model detected an ADC/DAC error and stopped driving the DAC outputs. Before restarting the models I pressed the DIAG_RESET button to verify the ADC error in the STATE_WORD cleared (it did) and the DAC error was latched on (it was). We also did a quick data dump of h1iopoaf0's proc status file and dmesg.

Here is the /proc/h1iopoaf0/status file before the restart

root@h1oaf0 /proc/h1iopoaf0 0# cat /proc/h1iopoaf0/status
startGpsTime=1162832563
uptime=5511
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162832990
adcHoldTime=15
adcHoldTimeEverMax=90
adcHoldTimeEverMaxWhen=1162837830
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=3
cycle=25821
gps=1162838074
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,5
cpuTimeMaxCycle(cur,past sec)=21,0
cycleHist: 3=65068@17 4=466@65535 5=2@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=17 Current=14
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0
 

The adcHoldTimeEverMax looks very high (we see it on other systems in the 60-70 range but not this high). The adcHoldTimeEverMaxWhen=1162837830 equates to Nov 10 2016 10:30:13 PST which is the time of the event.

On other systems running with 16bit DACs the FIFO_STATUS is always 2. After h1iopoaf0 was restarted, its status is now 2. Looking at the manual, it suggests

I've setup monitors to report on ADC and FIFO status if this happens again.

This is a continuity of 31381.

For a bookkeeping purpose, I have extended the spot position thing to PRM and MC mirrors. Here is a summary. They all seem fine as expected.

The measurement was done with the interferometer fully locked at 25 W in nominal low noise. The measurement precision should be something like 10% or so.

J. Kissel, D. Tuyenbayev

Following preliminary results from Darkhan on the individual actuation strength of the UIM and PUM stages for H1SUSETMY (see, thus far LHO aLOG 31275), and the current delightfully long lock stretch with them in place, I'm bringing this study to a close. I've turned off the temporary L1 and L2 calibration lines at 33.7 and 34.7 Hz, respectively. We do not intend on turning on these lines again for the duration of the run.

These lines were turned OFF at Nov 07 2016 21:21:49 UTC.