This morning Dave and Alastair reported one of the TCS AA chassis in the CER was found powered off. Switched unit back on and found the -15V led was flickering. Took chassis S1300387 to lab and found two buffer IC's (AD8622) had had failed on board S1300387 on CH3 and CH4. Replaced IC's and ran transfer functions. Unit was replaced back in rack.

07:51 Karen and Cris in the LVEA
07:58 Jim W. turning off HEPI to ISI feedforward on all BSC chambers to run measurement for ~ 20 min.
08:03 Hugh restarting end Y HEPI pump servo
08:15 Aidan going through the LVEA on the way to the high bay
08:18 Jeff B. to LVEA to look at dust monitors, work with Mitchel in West bay on bag and tag
08:21 Rick to end Y
08:40 Christina driving forklift from LSB to high bay
08:45 Sudarshan and Thomas to end Y for PCAL work
08:49 Cris to mid X
08:50 Alastair to LVEA to check on TCS laser
08:57 Aidan working on 3IFO near HAM6
09:01 Rick back from end Y
09:12 Betsy and Travis to the West bay
09:19 Andres moving 3IFO parts from the LVEA to the staging building
09:28 Doug to LVEA to take pictures
09:51 Richard to LVEA
09:56 Jeff B. and Doug out of LVEA
09:58 Jim W. running measurement on ETMY
10:00 Mitchel out of LVEA
10:03 Thomas and Nutsinee back from end Y
10:14 Travis and Betsy out of LVEA
10:45 Dave B. to CER to check on TCS AA chassis
10:51 Dave B. and Aidan to CER to power on TCSY AA chassis, does not work, Filiberto and Aidan investigating
11:37 Thomas and Elli back from end X
11:41 Thomas and Elli to end Y
11:42 Bubba to mid X to check on possible noisy air handler
11:54 Kiwamu starting WP 5047
11:56 Bubba back from mid X
12:09 Thomas and Elli back from end Y
12:16 Kiwamu restarting SUS ETMY model
      Bubba back to mid X
13:39 DAQ restart for new CALCS model
15:03 Jim W. to HAM5 to read serial number off box

model restarts logged for Mon 09/Feb/2015
2015_02_09 01:44 h1fw0
2015_02_09 02:28 h1fw1
2015_02_09 14:36 h1fw1

all unexpected restarts of frame writers.

model restarts logged for Tue 10/Feb/2015
2015_02_10 10:03 h1tcscs
2015_02_10 10:08 h1pemcs
2015_02_10 10:10 h1pemex
2015_02_10 10:10 h1pemey
2015_02_10 10:12 h1pemmx
2015_02_10 10:12 h1pemmy
2015_02_10 11:32 h1susetmx
2015_02_10 11:36 h1dc0
2015_02_10 11:38 h1broadcast0
2015_02_10 11:38 h1fw0
2015_02_10 11:38 h1nds0
2015_02_10 11:42 h1fw1
2015_02_10 11:42 h1nds1
2015_02_10 13:01 h1iopoaf0
2015_02_10 13:03 h1oaf
2015_02_10 13:03 h1odcmaster
2015_02_10 13:03 h1pemcs
2015_02_10 13:03 h1tcscs
2015_02_10 19:32 h1fw1
2015_02_10 20:23 h1fw1
2015_02_10 22:30 h1fw1

Maintenance Day. New code for TCS and PEM with associated DAQ restart. SUS ETMX restart to clear error. ADC timing glitch required restart of h1oaf0. Unexpected restarts of frame writer.

Attached are plots of coherence between DARM and other channels during last night's lock.

In the 100-200Hz region, there is some coherence with the other LSC DOFs, and the OMC ASC control signals (OMC-ASC_{ANG, POS}_OUT_DQ).

Above 300Hz there is coherence with the input intensity noise.

In the 20-40Hz band there is some coherence with the OMC ASC.

Below 10Hz there is a lot of alignment noise.

Question for DetChar people: does this agree with other coherence tools?

These are trends from the past 10 days

Dan, Evan

We measured the dark noise of the OMC DCPDs with the IMC unlocked and MC2 misaligned.  (We also trimmed the dark offsets.)

The attached xml file & plot has measurements of OMC-DCPD_{A,B}_IN1 and _OUT with zero, one, and two stages of analog whitening enabled (and the corresponding digitial dewhite filters).  The y-axis units are in cts/rt[Hz] for the IN1 channels, mA/rt[Hz] for the OUT channels.

With zero whitening (the case for the locks last night), the ADC noise is about 7x10^-3 cts/rt[Hz] at IN1.  Last night our noise was about 10x larger than this (see upper left plot here).

In response to yesterday's locking activity (alog 16628), I newly added the DARM->M0 damping path in the QUAD suspension models under WP 5047. This allows us for damping the bounce mode through the M0 stage using DARM signals as was done in LLO (LLO alog 16686).

(h1susi(e)tmex(y) models are modified)

Right now the modifications are completely local in the sense that I did not change the common QUAD master block but the local front end models, namely h1susetmx, h1susetmy, h1susitmx and h1susitmy. In these front end models, I disabled the link of the QUAD master block in the top layer so that they are now detached from the common QUAD master. I then added the DARM damping filters by coping the filters from Livingston's. I copied them from l1susetmx.mdl. Note that the latest-latest l1 quad suspensions are capable of routing the WFSs signals in addition to the DARM damping, but we don't have this at the moment.

I checked in all the four quad models into svn.

(some screens)

I then updated the medm screen, common/medm/quad/SUS_CUST_QUAD_OVERVIEW.adl (see the attached) which is a variant of what LLO currently has. I have not checked this screen into svn. Jeff will take care of it. Also I added a new screen, SUS_CUST_QUAD_M0_DAMP.adl which displays the filters. This is a duplication of what LLO locally has. Neither of the sites checked this screen in yet.

The attached PDFs show the resistance measurements for all RHs and the relative resistance vs time for the RHs as 1W is applied to each segment. All segments are operating within nominal parameters.

The resistances for the segments are:

J. Kissel,
 
I've updated the 
/opt/rtcds/userapps/release/sus/common/medm/
directory to receive a few overview screen and BIO screen updates that Stuart and Joe have made, see 
BSFM LHO aLOG 16368
QUAD LHO aLOG 16760


jeffrey.kissel@opsws3:/opt/rtcds/userapps/release/sus/common/medm$ svn up
Restored 'SUS_DRIFT_MON_SMALL.adl'
U    bsfm/SUS_CUST_BSFM_BIO.adl
U    bsfm/SUS_CUST_BSFM_OVERVIEW.adl
U    quad/SUS_CUST_QUAD_BIO.adl
Updated to revision 9804.

Turned off earlier to allow Thomas and Elli to enter end X.

BCS9 illumintator turned off at 11:30 PT.

Due to some extenuating circumstances, Travis, Richard, and I moved the remaining hardware out of the 2 large 16' assembly cleanrooms in the west bay of the LVEA and turned them off.  There is still a small cleanroom on against the West wall of the LVEA which we will turn off in a few weeks when we are finally finished kitting 3IFO hardware.

12 hour OpLev trends starting 5:51:04 UTC.  No obvious issues.

Just getting ranges correct and consistent for display of reds & greens.  All pressure lights are now Green and OK.  The SERVO database is set up so restart does not affect the HEPI platforms too much.  I of course would not do this if anyone was running any tests.  The times are: EY 1603, EX 1619, CS 1638utc.  Attached is 40 minutes of second trends with these times annotated.  The corner station pressure glitch was non existent, the ends had glitches of a few PSI.

When I look at the HEPI L4Cs during this pressure glitch, it looks like it is absorbed into the normal background glitching.  See the latter two attachments.

Sheila, Dan, Daniel, Evan Jeff, Lisa, Alexa

Starting at 08:43:06 UTC Feb 11, 2015 we had a DC lock for ~1:54min. We lost lock again because of the 9 Hz bounce mode in ETMY. We had tried damping with the L2 stage, which helped a bit but still ultimately caused the lock loss.  Tomorrow we should try damping with M0, as done at LLO (LLO#16686).

J. Kissel, B. Abbott

Over the past few weeks, I've been building up understanding of the HEPI pump servo -- more than I ever wanted to know. 

The conclusions from all this?
(1) The EPICs calculation of the PID loop (documented loosely in the EPICs manual here)
     (a) uses backwards differentiation approximation and velocity formalism to compute the *change* in control output value, which it then adds it to the previous cycle's control output value -- which turns it back into a "position" algorithm -- i.e. one *doesn't* have to differentiate the pole and two zeros of the P, I and D. Other good references I found are here, here, and here.
     (b) expects the "I" and "D" terms in [cycles / min] or [mins / cycle], respectively, i.e. it depends on the sampling rate / clock cycle turn-around time of the PID calculation. If you want to do any sane prediction of the filter shape, you've got to convert these to [rad/s] or [s/rad], again respectively, by multiplying by 60/(2*pi*Ts) [(min/sec).(rad/cycle)] or (2*pi*Ts)/60[(cycle/rad).(sec/min)].
(2) The EPICs turn-around time, or clock cycle, for the discrete PID controller calculation is longer than the requested sampling frequency, which means the sampling rate is determined by the clock-cycle. Further, it's slower in the corner station than the end stations. We should lower the EPICs record's demand of the sampling frequency from 10 [Hz] to 1 [Hz] (and check again if the PID can turn around the calculation fast enough).
(3) The HEPI Pump servo contains a second-order, 16 [Hz], Sallen-Key, anti-aliasing filter before the input to the ADC on all pressure sensor channels (see D0901559, pg 2)
(3) We should lower the PID parameters such that the UGF of the loop is ~ 1 [mHz]. Why?
     (a) The pressure sensors were never meant to be used as AC sensors. LHO aLOG 16500 hints that they shouldn't be used in a loop above a few [mHz].
     (b) The ADC noise of the Athena II, PC 104 computer is an abysmal 1e-2 [V/rtHz]. As they are currently amplified, the pressure sensor's signal gets buried in this ADC noise by ~10 [mHz].
     (c) Adding an EPICs "smoothing" parameter (a.k.a. SMOO) to the EPICs version of the pressure sensor channels adds a single-pole low-pass filter into the control loop. If sufficiently low in frequency, it'll start to creep in on your already-small phase margin. We should use this with caution, or at least be cognizant of its impact.

-------------
Details.

I attach 5 plots per pump station:

Pg 1: On the EPICs turn around time defining the sampling rate
In my perusing of the EPICs manual, I found that the PID channel, e.g. H1:HPI-PUMP_EX_PID, has sub records, one of which is "DT," which one can query with a simple caget:
jeffrey.kissel@opsws8:/$ caget H1:HPI-PUMP_EX_PID.DT
H1:HPI-PUMP_EX_PID.DT          0.312006
This is "the time difference in seconds between processing steps." Consequently, I caget'd this sub record 5000 times. This turned out to be faster than the EPICs calc record would update the number by 3, so I took every third report. Then I histogrammed the results, to find that the clock cycle is 0.55 and 0.28 [sec] (!!) for the corner station and both end stations, respectively. The end stations show some clock jitter, but I took the mode of the 1667 points and used that as the clock cycle. This immediately informs us that the features seen in all ASDs that happen at 1.8 [Hz] and 3.5 [Hz] at the corner and end stations are just a function of the terribly slow sampling rate -- even slower than the EPICs 16 [Hz], and the request calc record rate of 10 [Hz].

Pg 2: On EPICs Implementation Discrete PID Control
It's too difficult in a simple text editor to really do the explanation any justice, but check out all of the references I show above, while you wait for coherent presentation version of this aLOG. One of the many reasons why my initial guess at what the servo was doing (in LHO aLOG 16447) was incorrect was because I wasn't accounting to the time delay of the discretization. As I found out later, it turned out that the discretization was much slower that was defined in the EPICs calc record.

Pg 3: Modifying the measured plant (see LHO aLOG 16601) with the anti-aliasing filter, and an EPICs smoothing filter 
On the SMOOOOOOO at EY
Hugh's initial instinct to combat the newly-noisy differential signal was to add some EPICs sub-record defined "smoothing factor" to the supply and return channels which form the differential pressure signal. Again, from the EPICs manual, 
"The converted data value is subjected to the following algorithm:
val = newvalue * (1 - smoo) + oldvalue * smoo
SMOO should be a value between 0 and 1, with 0 meaning no smoothing and one meaning ultimate smoothing (in fact the data will never change)."
*sigh* who writes these manuals? Thank god they wrote out the algorithm. This is just the discrete realization of a first order low-pass filter. The pole frequency of which is defined by
f_{pole} = 1/(2*pi*Ts) * ln(1 / (1 / alpha))

Hugh had entered in a SMOO of 0.75, which at a dismal sampling rate of Ts = 0.28 [sec], means the pole frequency is at 0.159 [Hz], which explains *some* of the excess gain peaking that we see at EY. 

Pg 4: Loop Design figures of merit given the now-(mostly)-understood plant and controller

Pg 5: Model of closed loop ASD noise
Clearly I'm still missing some phase loss term that increases the gain peaking, but at least I get the suppression correct. I could hunt around for several more days as to why this is, and find some other nasty EPICs trickery. But I'm not gunna.

Attached is spectra from WHAM6 looking at the GS-13 INF IN1s--before the digital filters.

The Red traces where taken at 1417utc (~6am local) with the FM4 & FM5 ON, my understanding and based on the plots here, this is analog low gain and whitening.

The Blue traces are from 1550utc with FM4 ON and FM5 OFF--analog low gain but not whitened.

Green traces are from 1650utc with both FM4 & 5 OFF--analog high gain and not whitened; The brown traces start at 1722utc with FM4 OFF and FM5 ON, that is, analog whitening and high gain.

Again, based on the plots, the switches appear to be doing what I thought and expect.

Do we have saturations when in whitened high gain?  No, based on the H1:FEC-51_ACCUM_OVERFLOW, no saturations accumulated during any of the measurements.

What about being ADC limited in low gain without whitening.  Included on the traces are the ADC noise from the svn.  It certainly looks like in low gain without analog whitening, we approach the ADC limit at 100Hz and very likely about a few 100Hz.

Conclusion--Leave FM5 ON all the time--we are not saturating in high gain and we are near the ADC noise in low gain without whitening...but wait:

But what about those ugly lumps between 200 and 400Hz?  They do not show up on the unwhitened hi gain signal but are evident on the other three configurations.  The ringing is exactly 1Hz and the beating is about 45Hz.  Maybe this suggests a problem with the whitening filter?  Maybe we should not whiten the signal when in high gain.

I added a zoom in of the high frequency beat  stuff.

From JeffK's 16425, while I'm a bit confused as to Jeff's language I think i understand:

These two states are compensated for digitally in FMs 4 and 5 of the GS13INF banks, by the difference between the two states (the overall gain of 2 is folded into the calibration filter). FM4 is the switchable gain compensation, FM5 is the switchable compensating de-whitening filter. The front-end code for the HAMs and BSCs is set up that these digital banks control the analog switching. 
- When FM4 is ON, the gain switch is HI (or a binary output of 1), so the analog gain is 2, and FM4 compensates the gain of 10 difference. 
- When FM4 is OFF, the gain switch is LO (or a binary output of 0), so the analog gain is 20.
- When FM5 is ON, the analog whitening is LO (or a binary output of 0), so there is no whitening, and FM5 compensates the z:p = 50:10 difference. 
- When FM5 is OFF, the analog whitening is HI (or a binary output of 1), so the whitening is engaged.

and I think the FM5 states are backwards as to the above verbeage.  The digital DWH filter certainly looks like a De-Whitening and the above traces bare out the FM5 enables the analog whitening.

dtt template is in /ligo/svncommon/SeiSVN/seismic/HAM-ISI/H1/HAM6/HAM6_GS13_GAIN_DWH.xml

The first attachment is a time series of the HAM6 Watchdog, the Fastshutter State, and a GS-13 trace on HAM6.  The switch to low Gain is clear on this last trace.  This is a 5 day trend and the ~20 minutes needed to get the below .005Hz BW data is comfortably unmolested by the ISI or the fastshutter--the start times of the Spectra calculations begin at the vertical black lines.  As for ISI trips and shutter fires--there were none.

The High Gain Spectra period begins at 0800utc on 29 Jan, the Low Gain Spectra begins at 0800utc 1 Feb.

The second attached graph has Horizontal GS13s on HAM6 in High and Low Gain on the upper plot.  The Verticals are in the lower plot. High Gain Spectra is Dashed, Low Gain traces are Solid.  Also shown is our ADC noise in counts.

One caveat is that the ground motion is different at the two times.  The third attachment shows the Cartesian DOFs for the HAM6 GS-13s in the upper two plots and the ground noise in the bottom plot.  We've had a ground motion decrease in the microseism frequencies and below during the low gain period.  You can see the Low Gain Spectra are at the ADC level below 50mHz.  And the 10x signal reduction expected is more like 5x below these frequencies.  If this conclusion passes mustard, we'll have to revisit the GS13 triggering delays/levels.

DTT template is /ligo/svncommon/SeiSVN/seismic/Common/MatlabTools/HAM6_GS13_GainNoiseTest.xml