J. Kissel, Nutsinee

Dan found glitches in SR3 T2  last night and it seems like we have been having trouble locking ever since so we investgated. SR3 T2 problem started right before Aug 31 09:23 lockloss and never go away. DAC recalibration this morning didn't solve the issue and the power glitch didn't cause it. The noisy/glitchy feature looks just like what happened before the drive swap back in Aug 18.

The LHO SITEMAP MEDM was modified to add two new MAIN catagories: generic seismic (SEI) and Observing Run related screens (O-1).

I have added Hugo's generic SEI_SAT_OVERVIEW to the SEI button. I have added my new H1CDS_O1_OVERVIEW_CUST to the O-1 list. The new CDS OVERVIEW MEDM must always be GREEN. Any deviation from the O1 configuration will show as RED. Hopefully nothing will be red for long on this screen during O1.

h1nds1 (the default NDS) daqd process died this afternoon. This was a more drastic failure, the computer locked up and had to be reset. The last messages in the log files are different from yesterday to today:

Sunday 30 Aug 15:29:

Retry expired; requesting again

....

Packet buffer is full

Moday 31st Aug 15:58:

[date-time] Ask for retransmission of 15 packets; port 7097

....

Have to skip 15 packets (packet buffer limit exceeded)

The times of the restarts of h1sush2a and h1sush56 are summarized below.

2015_08_31 09:13 h1iopsush2a
2015_08_31 09:15 h1susmc1
2015_08_31 09:15 h1susmc3
2015_08_31 09:15 h1suspr3
2015_08_31 09:15 h1susprm
2015_08_31 10:09 h1iopsush56
2015_08_31 10:09 h1susomc
2015_08_31 10:09 h1sussr3
2015_08_31 10:09 h1sussrm

As per alog 21036, Evan dropped the absurd precision in the WFS_offset_* scripts which set the dark offsets.  I reran it now since the IFO is down, confirmed that none of the 80 offsets changed by very much, and then accepted them in SDF.  Now at only 4 sig didgits, SDF was able to write them into the SAFE.snap when I hit ACCEPT all.

ER8 Day 14

model restarts logged for Sun 30/Aug/2015
2015_08_30 15:30 h1nds1
2015_08_30 15:31 h1nds1

both restarts unexpected.

J. Kissel, K. Izumi, S. Dwyer, N. Kijbunchoo

While trying to being the h1sush56 SUS (SR3, SRM, and OMC) to safe for DAC calibration of all cards in that chassis (see LHO aLOG 21048), we had great trouble with the SUS_SR3 guardian node. This node is not managed, but it was entire unresponsive to requests to change its state, to load, to pause, to stop, anything. Sadly, this was all true and the guadian node screen did *not* turn red indicating its in error.

Looking at the guardian log, there was a message:

epicsMutex pthread_mutex_unlock failed: error Invalid argument
epicsMutexOsdUnlockThread _main_ (0x3e7a0f0) can't proceed, suspending.

This is after requesting the ISC_LOCK guardian to DOWN. Note, we also had the SR3_CAGE_SERVO guardian still running, because, though this is managed by the ISC_LCOK guardian, it does not turn it OFF in the DOWN state. Probably not the issue, but it gathered our attention because it had gone nuts and was driving the M2 stage into constant saturation.

See first attachment for screenshot of the broken situation.
For the record this has happened on a smattering of guardian nodes in the past, see LHO aLOGs 17154 and 16967.

-----------------------
Here's what we did to solve the problem:
- Tried to restart the guardian node from a work station,
guardctrl restart SUS_SR3
No success.
- Tried to destroy the guardian node from a work station,
guardctrl destroy SUS_SR3
No success. 
Both report
stopping node SUS_SR3...
timeout: run: SUS_SR3: (pid 3042) 23232015s, want down, got TERM
(see second attachment).

- Tried logging into guardian machine, h1guardian0, and began to kill process IDs sound on the machine related to SUS_SR3, (see third attachment).
- Curiously, as I killed the first two processess,
runsv SUS_SR3 and
svlogd -ttt /var/log/guardian/SUS_SR3
as soon as I killed the latter, the guardian log came alive again, and the the SUS_SR3 node became responsive.

- At Kiwamu's recommendation, I killed all processes simultaneously, destroyed the node, and restarted the node, just to make sure that all bad joojoo had been cleared up.
----------------
The problem is now solved, and we've moved on.

Unsure what to do about this one in the future other than the same successively agressive restarting techniques...

This is a follow-up on alog alog20541 and alog20717

I took some time to study Robert's noise injections in more details. The result attached below. Quick conclusion: Human jumps in the change room and car sudden breaks near 2k electronics building and high-bay seems to couple into DARM.

re aLOG 20296, the bleed off of the tidal drive to the HEPIs tilts the platform enough to rile up the ISI T240s.  Looked at trends for the last 30 days; the bleed down rate changed from 2 to 1u/s on 6 August.

Since then there have been many lock losses where there was some tidal buildup to be bled off.  The diagonalization of the EndX HEPI must be pretty good and gives no indication of tilting the T240 much.  Unlike X, the HEPI for EndY shows some fairly substantial tilting as the build up is reduced.  See the attached for 15 minute plot of a recent Bleed off at EndY where the ~110um bleed off at 1u/s riled up the T240s more than half way to the trip point.

The EndY HEPI could use some diagonalization

No indication that Accumulators need checking or charging.

Trends of Pump output pressures (attached) indicate no need to restart any pumps (no DC level changes pointing to cavitation.)  Yes those are actually pressures; I don't know why the trends of the L0 (corner station) don't show any more fluctuations like they do at the Ends...  No change at EndY, same daily and weekly glitches.

No further HEPI maintenance this week.

Peforming some calibration measurements in the past weekend, I noticed that the red TRX and TRY QPDs had some dark offsets which were not quite cancelled. In this morning, since the interferometer was down, I adjusted the dark offset in each segment of all four qpds (ASC-X_TR_A and , and ASC-Y_TR_A and B). Subsequently I adjusted the LSC-TR offsets which receives the sum signals from these qpds -- the offsets are now zero as expected.

At the request of Richard M and Nutsinee, I restarted the models on h1sush2a to force a recalibration of the 18bit DAC cards.  All cards reported successfull calibration.

Dan, Jim

The IFO was running smoothly in low-noise for almost four hours until a lockloss at 0923 UTC.  Immediately prior to the lockloss, Jim and I noticed some motion of the AS beam spot.  There were a few bursts of motion, a few minutes apart, and then we broke lock.

The first image is a 1-hour trend of the ASC-AS_C signals (channels 7 & 8).  There are some bursts of noise in ASC-AS_C_PIT, correlated with similar noise in the M3 witness sensors of the SRC optics.  SR3_WIT_P (channel 15) is the worst offender.  This is the error signal used for the cage servo.  The control signal is SR3_M2_TEST_P (channel 10), which has some excurions from the quiescent level at the time of the noise bursts.

So, it looks like something was kicking the SR3 OSEMs, and this got into the cage servo and broke the lock.  Since nothing feeds back to SR3 it must have been an internal problem.

The second plot is a 1-hour trend of the SR3 OSEMs.  M1_T2 stands out as a problem, and noise from this sensor would couple to pitch in the lower stage.  Channel 16 is the T2 VOLTMON; clearly T2 is having issues.  This is the same coil that Evan, Jeff and I had trouble with two weeks ago.  Since then, the SR3 coil driver was swapped.  The glitches don't have the same shape as the excusions back then.

Jim and I have been watching a dataviewer trend of the T2 voltmon and there are occasional large excusions that are correlated with the M3_WIT_P channel, so something is really moving.  We held the ouput from the cage servo and saw a few more glitches, so it must be coming in from the top stage damping.  We turned off the top stage damping and we *thought* we saw a few more, so it must be coming from the actuation electronics.  The problem is very intermittent and we're not totally confident in the diagnosis.

Btw the T1 voltmon is still busted.

Vern asked for some of the pie charts I made for ER7 for the first weeks of ER8.  Here they are, based on data from 15:00 UTC August 17th to 15:00 UTC on the 28th, with about 40 minutes of missing data.

A few notes:

• We are suceeding in the CARM offset reduction more often. Durring ER7 only 1 in 5 DRMI locks reached the nominal operating state.  Now 1 in 3 DRMI locks reach the low noise state, and 75% of them make it through the CARM offset reduction.
• We have locklosses due to the beamsplitter coil driver switching (there is a transient due to analog switching), locking since this guardian state uses blocking calls these locklosses are listed in the pie chart as Low noise ESD ETMY, and probably account for most of the 11 locklosses from this state.  This is a lockloss we did not have durring ER7 since we did not switch the BS coil drivers.
• We also had a series of locklosses from the state DRMI on POP.  Durring this state we switch the state of the PRM coild driver.  While this coil driver was causing huge glitches we took one coil out, in this condition the switching became unreliable and caused locklosses.  This problem has been solved by replacing the chasis. 
• In the first few days of the run we had difficulty with SRC ASC, which caused many locklosses from the states DC readout, Increase power, and other states neaar this part of the sequence.  This is a problem that has been partially addressed, but will likely return.  We hope to spend some commisioning time to better understand the phasing and sensing matrix for these AS WFS.  It would be good to do this now, while the ASC is working.
• Although we have many locklosses from the earliest steps of the locking sequence, Locking arms green and Locking ALS, these locklosses are not much of a concern since they do not cost us much time.
• DRMI locking times are similar to ER7, the median time to lock is 4 and a half minutes (It was 5 minutes durring ER7).  We still have some extremely long locking attempts (up to 2 hours!)  I'm hoping most of these were commisioning. 

Our duty cycle is a bit worse than it was durring ER7, although there are several reasons we would expect it to have improved.  Some of the downtime can probably be explained by commsioning and calibration activities, althoug I haven't yet tried to take that into account.  It will be easier to get a clear picture of things once we start running undisturbed more. 

Nairwita Mazumder, Rich Abbott

A few days back Jim noticed  (alog ) that the "Bumbling line" which varies over a large frequency range is again back on ETMX seismic channels . This was first noticed on March and disappeared before ER7 and again was seen from 4th August. One can see the lines at all the horizontal and vertical sensors on ETMX. 

I have attached a pdf containing some follow up work done during Rich's recent visit to LHO. The first plot in the pdf is the spectrogram of ETMX GS13 on 26th August. It can be seen that there are multiple wandering lines having a fixed offset.  We were suspecting that some magnetometers at the End X might be the culprit (as we could not find any correlation between temperature fluctuation  with the line ). 

The second and third plots are the spectrum of H1:PEM-EX_MAG_EBAY_SEIRACK_Z_DQ and H1:ISI-ETMX_ST2_BLND_Z_GS13_CUR_IN1_DQ for 2nd August and 26th August respectively. The red one is for 2nd August when the bumbling line could not be found and the blue one is the recent data (26th August). It is clear that the peaks appearing on  ISI-ETMX_ST2_BLND_Z_GS13 after 3rd August are correlated with the peaks of the spectrum (which also appeared around the same time) of SEIRACK magnetometer .

The plots on the second page shows the coherence between GS13 and the magnetometers in the VEA and SEIRACK. It looks like  the magnetometer on the SEI rack has stronger coherence with GS13 sensors than the magnetometer located at VEA .  I have marked two points (blue and red cross)  in the coherence plots to highlight two of the many peaks.

I have completed a set of measurements that is necessary for nailing down the ETMY actuation scale factors. This is a third round of the measurements in order to study repeatability and systematic errors. The measurement data can be found at the follwong places.

• Full lock ETMY L3
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_L3toDARM_LVLN_LPON_FullLock.xml
• Full lock ETMY L2
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_L2toDARM_FullLock.xml
• Full lock ETMY L1
  • /aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_L1toDARM_FullLock.xml
• Full lock ETMX
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMX_toDARM_FullLock.xml
• Full lock Pcal Y sweep
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_PCALYtoDARM_FullLock.xml
• Full lock Pcal Y sweep in wide band (7 - 1200 Hz)
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/PCAL/2015-08-29_PCALY2DARMTF_7to1200Hz.xml
• Full lock DARM open loop in wide band (7 -1200 Hz)
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/DARMOLGTFs/2015-08-29_H1DARM_OLGTF_7to1200Hz.xml
• ALS Diff ETMX drive
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/ALSDIFF/2015-08-29/2015-08-29_ALSDiff_ETMX_L3_HVHN.xml
• Mich lock ITMX drive
  • /aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1SUSITMX_L2_State2_MICH.xml
• Mich lock open/closed loop
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_MICH_OLGTF.xml
• Mich free swing time series
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1MICH_freeswingingdata.xml
• Xarm locked ITMX drive
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1SUSITMX_L2_State2_XARM.xml
• X arm locked ETMX drive
  • aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1SUSETMX_L3_HVHN_XARM.xml

Kiwamu, Nutsinee

As we were trying to relock the ifo after several locklosses due to high wind (50mph), we noticed the sideband signals wiggled a lot before another lockloss at DC_READOUT (wind speed ~35-40 mph). We found a coherence between POP18, POP19, POP_A_LF, AS90 and PRM, SRM, BS which indicates that the DRMI was unstable. The BS ISI Windy blends weren't turned on. 

J. Kissel, K. Kawabe, S. Karki

We've broken observation mode such that we can enable the DAC DuoTone timing readbacks on the front ends that are responsible for DARM control, i.e. h1lsc0, h1susex, and h1susey. We needed to take the IFO down for this because the last channel on the first DAC cards for the end station SUS are used for top-mass OSEMs for damping the suspensions. If the damping loops get a two sign waves at 960 and 961 [Hz] instead of the requested control signal for one of the OSEMs, then we get bad news.

Here are the times when the DAC DuoTone switches were ON for the following front ends:

h1susex and h1susey --- 19:04 to 20:04 UTC (12:04 to 13:04 PDT)
h1lsc0 --- 19:16 to 20:06 UTC (12:16 to 13:04 PDT)

Though all relevant channels (ADC_0_30,  ADC_0_31, DAC_0_15) are free on the h1lsc0 front end, we elected to turn the DAC DuoTone off, so that we aren't in danger of an oscillitory analog voltage being sent around the IO chassis that's used to measure the OMC DCPDs.

Data and analysis to come. 

The IFO will be staying down for a few hours, while we finish up some electronics chain characterization of the OMC DCPD analog electronics (along with some other parasitic commissioning measurements).