Cheryl emailed me overnight to mention that GWIstat was reporting an incorrect status for LLO -- it was reporting "Not OK" (red) while LLO was actually observing (OK+Intent).

I investigated and found that this was due to additional bits that became active recently in the GDS-CALIB_STATE_BITMASK and are being (correctly) summarized by Chad Hanna's gstlal processes, which my gwistat_collector process polls.  When I wrote the gwistat_collector parsing code during ER7, I had assumed that only 9 bits of GDS-CALIB_STATE_BITMASK were going to be used.  Maddie announced upcoming changes to the h(t) calibration channels and expansion of the bitmask on August 24, but it didn't register with me that it would affect GWIstat.  Scanning the hoft frame files, I see that the extra bits were turned on starting on August 28 at both sites.

So, the GWIstat status display has been generally incorrect since August 28, but is fixed now.  I apologize for the confusion.

For reference, here's how I checked to see when the extra bits first appeared:
pshawhan@> hostname -f
ldas-pcdev1.ligo.caltech.edu
pshawhan@> cd /archive/frames/ER8/hoft/L1
pshawhan@> /home/pshawhan/hwinj/monitor/FrBitmaskTransitions -c L1:GDS-CALIB_STATE_VECTOR -m fe00 L-L1_HOFT_C00-1124[89]/*.gwf 
1124802560.000000  0x00000000  Data starts ( )
   *** *** ***                 Data gap from 1124833796 to 1124833820
1124833820.000000  0x00003e00  9 on, 10 on, 11 on, 12 on, 13 on
1125003264.000000  0x00003e00  Data ends ( 9 10 11 12 13 )

Dan, Daniel, Evan

Summary

The addition of a 9.1 MHz bandpass on the OCXO output has removed the broadband excess noise between DCPD sum and null. The dashed lines in the figure show the sum and the null as they were three days ago (2015-08-31 7:00:00 Z), while the solid lines show the sum and the null after the filter was inserted.

Details and review

Since at least June (probably longer), we've had a broadband excess noise between the sum and null DCPD streams. Stefan et al. identified this as 45.5 MHz oscillator noise a few weeks ago (20182).

In parallel, we switched the 9 MHz generation from an IFR to the OCXO (19648), and we installed Daniel's RFAM driver / active suppression circuit (20392), but the excess noise remained (20403). For a while we suspected that this was 45.5 MHz phase noise (and hence not supressed by the RFAM stabilization), but the shape and magnitude of the oscillator phase noise coupling (20783) were not enough to explain the observed noise in the DCPDs, under the assumption that the OCXO phase noise is flat at high frequencies (20582). For that matter, the shape and magnitude of the oscillator amplitude noise coupling were also not enough to explain the observed noise in the DCPDs, assuming a linear coupling from the RFAM (as sensed by the EOM driver's OOL detector) (20559).

Daniel et al. looked at the 45.5 MHz spectrum directly out of the harmonic generator in CER, and found that most of the noise is actually offset from the 45.5 MHz carrier by 1 MHz or so (20930), which is above the bandwidth of the RFAM suppression circuit. This suggested that the noise we were seeing in the DCPDs could be downconvered from several megahertz into the audio band.

Yesterday there was a flurry of work by Keita, Fil, Rich, et al. to find the source of this excess noise on the 45.5 MHz (21094 et seq.). Eventually we found circumstantial evidence that this excess noise was caused by baseband noise out of the 9.1 MHz OCXO.

Tonight we installed a 9.1 MHz bandpass filter on the OCXO output. This has removed the huge 1 MHz sidebands on the 45.5 MHz signal, and it also seems to have greatly lowered the coherence between DCPD A and DCPD B above a few hundred hertz.

Loose ends

The chain from OCXO to filter to distribution amplifier currently involves some BNC, since we could not find the right combination of threaded connectors to connect the filter to the amplifier. This should be rectified.

Also, it appears that our sum is lower than our null in a few places (400 Hz in particular), which deserves some investigation.

I have been blaming the MICH Pit ASC loop now for a while for some locklosses that we've been seeing during the CARM offset reduction steps, specifically around the Switch_to_QPDs step (cf alog 20988).  However, when looking at MC2 for the glitch patterns from last night, I saw that perhaps we are saturating MC2's M2 coils and losing lock as a result.  

Attached below are 4 examples of this lockloss type from today.  Normally, the MC2 M2 Master out channels are well below 10,000 cts peak-to-peak, but just prior to lockloss they start to grow, and then cross over the 130k DAC count limit.  Since the IMC is near the core of all loops relating to common mode things, I still have not disentangled why this is happening, but maybe we can work on preventing it from being a problem.

Shift Timeline:

00:45 - drop out of observing - PEM injections and violin damping resume

05:01 - went to observing

06:02 - forced lock loss for tests

Currently:

I had to align both TMSX and TMSY to get the green arm powers above 1.  Possibly related to the very nice 12+ hour lock we had.

Sheila's changes loaded

Jim is relocking

Tasks handed off from Corey:

- sheila, complete changes from last tnight

changes - guardian line uncommented

filters - changed from ramp to immediate

--- reloaded 07:01UTC

- gerardo, watch the voltages, should be about 7kV on HOVE_MY.adl - still at 7kV, handed off to Jim

- Robert and Anamaria - OEM injections - done now

After making a few "OBSERVE.snap" files for a few SDF nodes during the DAY shift, I tried taking the nodes back to their "safe.snap" files, but some of the nodes ended up having differences!  Since we need to have ZERO differences to allow going to OBSERVATION Mode, I did what I could to make the differences as close to a "safe" state as possible.  For the "safe" files which had differences, I looked used whatever .snap file which got us as close to GREEN as possible. 

Here are the Nodes I worked on, the .snap file they are currently running, and a note about differences observed:

    1.    SUS-ITMX:  safe & GOOD!——> but now has nutsinee differences!
    2.    SUS-ITMY:  safe & GOOD!
    3.    SUS-ETMX *:  safe_150902_124725 & (1) Diff:  H1:SUS-ETMX_L2_DAMP_MODE1_GAIN ???, set point 0 & currently 50—>violin 1004.54Hz
    4.    ISI-ITMy:  safe_150902_124924 & GOOD!
    5.    SUS-ETMY:  safe_150902_151039 & GOOD!
    6.    HPIITMY **:  safe_150810_115155 & (1) Diff (H1:HPI-ITMY_OUTF_H2_TRAMP?? , set point 60 & currently 5)—accept
    7.    ISIBS:  safe_150902_151938 & GOOD!
    8.    HPIBS:  safe & GOOD!

* The SUS-ETMx difference was a gain change Nutsinee made.  She returned the gain to 0.

** The HPIITMY difference was a Time Ramp, so I ACCEPTED.

To damp ALL 8 frequencies of the ITMX first harmonics.

MODE1 damps 996.53

MODE7 damps 995.36

MODE8 damps 995.64

MODE9 damps 992.43, 992.79, 994.28, and 994.73

MODE10 damps 996.25

MODE1, 7, 8, 10 MUST stay on if MODE9 is on.

The damp settings have been added to the table.

One of the issues that has caused us a few locklosses over the last several days is a transient that happens when we engage a boost in DHARD P. 

The solution that has worked at least once today was to comment out the engaging of the boost in the state CARM 5 pm, and engage it latter in the sequence when the loop has higher gain(21133) .  However, this is the boost that has saved us from the DHARD pit instability in the refl trans state that plauged us durring ER7 (19243) and for a long time before that. We would like to engage it before transitioning the CARM control to the refl sensor to avoid the return of this instability.

I downloaded some data of transitions through the CARM 5 pm state.  In the attached screenshot the red traces are from locking attempts which failed either in this state or the next one, and the blue traces are from locks which survived. It seems as though this is a transient from engaging the first of the boosts, which had zero history and a 0.1 second ramp.  The impulse response of this filter is about 2 seconds long, and the DC gain is about a factor of 2. 

Evan and I set this filter to zero history, immediately, and also changed the 23Wboost in DHARD pit, from zero history 0.1 second ramp, to zero history immediately. 

We also uncommented line 1321 in the ISC_LOCK gaurdian, ezca.switch('ASC-DHARD_P', 'FM1', 'ON'), so that this filter will be engaged before the transition of CARM to the refl sensor. 

Cheryl will load the new filter and reload the ISC_LOCK gaurdian if we unlock, and knows how to undo the changes if necessary. 

Confirmed with a trend that the ambient temperatures have pushed the EX chiller into and past the low alarm levels.

The Major/Low alarm at 23:42UTC had a temperature of 34.99F.

The channel went back in the NO_ALARM state at 23:47UTC with a temperature reading of 35.45F.

I followed the alarm handler guidance and made a 10 day trend which shows that this has been happening for a while, so no notification/action needed.

J. Kissel, C. Gray, B. Weaver

Every once and a while while Betsy and Corey are "beating the streets" to find out why there are SDF differences between the settings definition file ("safe.snap") and the current value of the EPICs record for every front-end model, they ask me about differences in the CAL-CS model. Often times, including this time, there are hardware injection channels that show up, simply because the INJ infrastructure lives in the CAL-CS model. Over time, I've been changing these channels to NOT MONITORED, because they all appear to be EPICs records that are constantly updated with information about the last injection, or some recent GRB alert, etc. I attach a screenshot of this list as it currently stands.

For convenience I type the list here and give it the 'ol college try to explain them:
H1:CAL-INJ_EXTTRIG_ALERT_QUERY_TIME < a numeric GPS time that changes any time when TINJ (the hardware injection scheduling software) queries new external alert occurs
H1:CAL-INJ_EXTTRIG_ALERT_TIME       < a numeric GPS time that changes whenever there is a new external alert
H1:CAL-INJ_EXTTRIG_ALERT_SOURCE     < a string record from what telescope the external alert came (SWIFT, FERMI, etc.) that changes whenever there is a new alert
H1:CAL-INJ_EXTTRIG_ALERT_ID         < a unique alpha-numeric record for each record that changes whenever there is a new alert (e.g. E156669)
H1:CAL-INJ_EXTTRIG_ALERT_TYPE       < a string record indicating the claimed astrophysical source of the alert (e.g. GRB) that changes whenever there is a new alert
H1:CAL-INJ_TINJ_ENDED               < a numeric GPS time when the last hardware injecton ended
H1:CAL-INJ_OUTCOME                  < a numeric code labeling the level of success of the last hardware injection
H1:CAL-INJ_TINJ_START               < a numeric GPS time when the last hardware injection started
H1:CAL-INJ_TINJ_STATE               < a status vector of TINJ
H1:CAL-INJ_TINJ_TYPE                < a string indicating the type of last injection
As I understand it, all of the EPICs records are essentially readbacks, and therefore should not be monitored by the SDF system.

You can peruse T1400349 is you'd like more details on TINJ and the hardware injection infrastructure. T1500197 is the document for the external alert system.

WP 5455

A new frame writer, h1fw3, has been added to the DAQ system.  This frame writer is a disk clone of the frame writer that Livingston is running, with the system identity changed to LHO h1fw3.  The purpose of this frame writer is to see if there are differences in retransmission request rates between h1fw2 and h1fw3.  The h1fw3 is writing science, commissioning, and trend files to a local disk.  The raw minute files are not being written by h1fw3. 

If all tests are successful, the new frame writers will be installed in a permanent configuration on Tuesday Sept. 8.

9/2 DAY Shift:  15:00-23:00UTC (08:00-16:00PDT), all times posted in UTC

Summary:  This morning we had an unhealthy H1.  The first few hours were devoted to troubleshooting H1 & solutions found (see Kiwamu's alog).  H1 has been running for over 6hrs (but with fairly big glitches) & has been in/out of Observation Mode.  It looks like Hardware Injections have been running most of the shift.  PEM injections started around 1:20pm.

Had some issues saving .snap files from the SDF interface.  Have talked to Betsy & Jamie about it.  RED SDFs were GREEN-ed, and I will post an alog with notes on this.

Support:  Jenne handed off to Kiwamu & then Sheila/Betsy later joined.

Activity Log:

• H1 Investigations
• 15:53 Solar Panel work on grass out back (Filiberto, Richard, Gerardo)
• 17:37  H1 Taken to Observing Mode
• 17:55 Entrance to Elec Bay (CER) room
• 18:55 Pushed out of science mode when Betsy & I made SDF changes
• 19:54 Went back to Observing Mode
• 20:20 PEM injections begin by Robert/Anamaria in LVEA (taken to Commissioning mode)
  • 21:05 Now at EY
• 21:27 Gerardo investigating VAC issue at MY
• 22:28 Barker loaded filters for ETM PIs (because they have RED CFC on CDS Overview

I have created a simple H1IFO_RANGE.adl MEDM screen. This is mainly so users can get the channel name of Jonathan's IFO range epics channel. It is linked to the sitemap under the O-1 button.

To better manage the H1 IFO configuration changes during ER8 and into O1 I have committed all pending modifications of source files, filter files and safe.snap files into SVN.

Turns out I was the main culprit for model changes not being committed, the changes I made yesterday to odcmaster and pemcs had not been committed, nor my change some time ago to calex to add the irig-b channel to the science frame. I have committed these today.

The filter files for the new PI models were still static files in the chans directory. I moved them to the userapps/sus/h1/filterfiles area, made the symbolic link and committed them to SVN. The CFC bit was raised during this shuffle, so the DIAG_RESET button for both PI models were pressed to clear this bit.

All remaining local mods to filter and safe.snap have been committed with a generic "version as of 02sep2015 ER8" message applied.

J. Kissel, D. Tuyenbayv

After a quick look at the results from the UIM driver measurements taken for calibration purposes (see LHO aLOG 20846), and hearing similar whisperings of confusion from Darkhan in his detailed fitting analysis (progress continues, aLOG pending), I got the chance yesterday to quickly take a look myself to confirm my initial suspicions about the FAST I MON response to DAC drive measurements we took, and how they're "missing" the 85:300 zero:pole pair; see LHO aLOG 21127. I drove myself nuts for a half-a-day trying to reconcile the overall gain on the UL coil with what was expected to be sure *that* was right -- assuming it would provide clues if I couldn't reconcile it. I couldn't. 

I subsequently measured the respose of all UIM drivers of all QUADs and confirm that I was not insane. While the jury is still out on my sanity, at least I can confirm that
- The FAST I MON / TEST L EXC response for all four coils on all four of the Modified UIM Drivers are missing the expected 85:300 Hz zero:pole pair 
  EDIT: This is because the FAST I MONs are immune to this frequency response. See Below.
- This measurement techniue reports a factor ~2 less drive strength on the ETMY UL and LR channels, likely a busted monitor board.
- ITMY's UR and LR monitor board signals are total hosed.
I'll update the Integration Issue #9 with the later two bits of information.

EDIT: Why are the FAST I MONs Insensitive to the 85:300 [Hz] zero:pole pair? Turns out I just can't follow my own math. As it clearly states in My Notebook, you need to divide the current monitor by the output impedance -- i.e. whatever's in-between the voltage monitor pick-off and the coil current pick-off. Hence my equation I copied in LHO aLOG 21127,
                  R24_{MON}       1                   1     1       1      
DC calibration  = --------- x ---------- x G_{ADC} x --- x --- x ------- 
                  R25_{MON}   2 R5_{UIM}             E2O   CBG   G_{DAC} 
is wrong, and it should be
                  R24_{MON}       1                   1     1       1    
DC calibration  = --------- x ---------- x G_{ADC} x --- x --- x ------- 
                  R25_{MON}    2 Z_{OUT}             E2O   CBG   G_{DAC} 
where Z_{OUT} is formed by the whole R4, R5, and C12 ( = R27, R23, C26) network. This network is from where the 85:300 [Hz] zero:pole pair originates. Once I divided the transfer function with the right impedance -- voila! -- the 85:300 zero:pole pair appears.

The message again: The FAST I MONs are immune to the very frequency response we want to measure. So is every other monitor circuit, so we *have* to measure these drivers using analog electronics if we want to characterize this pole zero pair with any high precision than calculating what it should be from the component values. That being said, I don't think we'll time for this. In its stead, in case we don't find time, I checked the relevant resistor values inside spare modified UIM driver, and they we *at worst* 0.3% discrepant for all four channels on the spare. It's *very* regrettable that the monitor board appears to be giving discrepantly low DC transconductance, because that begets suspicion that the compenents are wrong, but I would be MUCH more inclined to blame the monitor board than the actual coil driver circuit given that the coil balancing gains are so similar, and using the stage hasn't brought up any issue.

Importance of this in the big picture: this re-opens the mystery of the frequency-dependent residual seen between naive model and measurment in the early results of the UIM actuation strength in LHO aLOGs 21015 and 21049. 

Stay tuned for Darkhan's results fitting the coil driver response poles and zeros that *are* there. After O1, we'll look into perfectly compensating all of these electronics, so we never have to think about this again.

[JimW, Jenne, Evan, DanH]

Relocking after we took the IFO down for some Cal measurements about 7 hours ago has been pretty painful.  However, we're back up.  We were back up, before DHARD Pit kicked us out.

Our biggest problem was with some glitches that we think are in one of the mode cleaner mirrors.  They went away after a few hours though, so we aren't really clear on what the problem is.  We see them in MC2 MasterOut channels, so the glitches do exist in the digital system.  But, loops are closed everywhere, so it's tricky to tell where they originate.  While these glitches were present, we couldn't lock ALS for more than a few minutes at a time.  We would lose the lock before we were able to lock the DRMI, much less transition off of ALS.  The 2nd and 3rd attachments show quiet and noisy times, respectively.

Later, we lost lock a few times at Switch_to_QPDs.  Last month, Sheila and I kept adjusting the TRAMP on the reduction of the CARM offset that happens in that state, so I changed it again tonight.  I still don't have any particular understanding of why the speed at which we reduce the CARM offset matters so much.  For a while, we had been making it longer and longer (up to a Tramp of 60 sec), and then eventually reverted to the original 10 second ramp time.  Tonight, I put in the guardian that it should do this ramp over 30 seconds, and it's worked a few times in a row.

Next up was a set of weird PRCL glitches.  The attached lockloss screenshot shows them appearing in PRCL out, and the effect they have on the power recycling cavity's 9MHz buildup.  Some time around here, although I don't remember the exact order of operations (Evan, do you remember?) Evan reverted the 9MHz RF source from the IFR (which he installed during the calibration break earlier in the evening) back to the OCXO.  I don't know if we have a way to tell, other than someone's memory of the time, whether we were on the IFR or the OCXO at the time of this PRCL weirdness.  Anyhow, I think this only caused one lockloss, and it doesn't seem to be there right now. 

We have been seeing DHARD pitch oscillations during the CARM offset reduction for the past few days now.  Sheila and I were going to do some loop measurements last night to see why the loop was so unstable, but last night had its own troubles.  Anyhow, Evan commented out the turn-on of DHARD pitch FM1, "boostLTE".  This boost increases the gain below 1 Hz, but eats about 70 degrees of phase at 1 Hz.  With this commented out, we were finally able to get to DC Readout.  After we were sitting on Nominal Low Noise for a while, DHARD Pit started oscillating.  We saw it in the ASAIR camera, and in the ASC control signals.  My current hypothesis (without doing any measurements...) is that the IFO isn't stable enough for us to handle this semi-delicate loop before the rest of the ASC comes on.  So, I have added FM1 of DHARD Pit back to the guardian, but I put it at the very end of Engage_ASC_Part3 so that it comes on last.  Since it didn't ring up for more than ~30 minutes, I think delaying the boost turn-on should be okay.

The last sticking point didn't cause a lockloss, but it did prevent the guardian from getting to Nominal low noise on its own: The IMC guardian got stuck trying to turn on the ISS.  The ISS guardian has some logic that if the absolute value of the 1-second average of some number (PSL-ISS_SECONDLOOP_SIGNAL_OUT16) is above 2.5, don't turn on the ISS.  If this happens, it returns to the Locked state, but the ISC_Lock guardian is still requesting ISS_ON, so I think it should go back to trying to preparing the ISS, and rechecking this number.  Instead though, it just hangs at the Locked state.  I tried hand-requesting ISS_ON a few times, with the same result.  I don't know what this channel means, and I don't know how conservatively that threshold was set, but to try to get the lock to complete, I temporarily changed the threshold to be 3.5.  (After we got past this state, I put it back to 2.5 and reloaded the guardian, so next lock it will be back to the original threshold).  What is this signal (it's totally unclear from the screens, and I haven't opened up the model to see if I can figure it out)?  What are the consequences of increasing this threshold?  Why is the guardian getting stuck?  I think we need to re-examine this guardian logic. 

In this morning, I checked the OMC DCPD electronics chain (for both A and B) by injecting known sine wave analog signal. This was one of those items that Keita suggested me a while ago.

According to the data, I am concluding that our calibration model needs to add another pole at 10-ish kHz for accurately simulating the OMC whitening circuits.

Method

The measurement method is straightforward -- it is a swept sine measurement in a manual way.

I had a function generator by the HAM6 electronics rack which drove a single-ended-to-differential convertor (D1000931, technically speaking this is a coil driver test box). Then the differential signal is sent to the input of the OMC DCPD whitening board (D1002559, S1101603) by some clipping technique. By the way, the actual cable for connecting the OMC DCPDs were unplugged during the measurement. The excitation amplitude was set to 2 Vp-p at the function generator which resulted in 2 Vp-p in both positive and negative paths at the input of the whitening board as expected accroding to the schematic of the coil driver test box.

I then recorded the data in IOP at the full sample rate using dataviewer for 1 sec for a selected excitation frequency (and for some reason, diaggui did not want to run and hence dataviewer this time). Keeping the same excitation amplitude, I manually stepped the freqyency from 8 kHz to 100 Hz. After every step, I saved the time series of the IOP so that I can make a transfer function later. In addition, I had an oscilloscope with me which kept monitoring the excitation ampitude at the input of the whitening board. The scope told me that the excitation ampltude stayed constant at 2.02 Vp-p in each channel throughtout the measurement. The OMC DCPD had

Analysis and result

To get a transfer function from the data that I took in time series, I decided to do a sine-wave fitting for each data chunk to get the amplitude information. I wrote a small matlab script to do it. It can be found at:

aligocalibration/trunk/Runs/ER8/H1/Scripts/OMCDCPDs/Matlab/OMC_DCPD_AnalogChain_TimeSeries.m

The script utilizes fminsearch and spits out the best fit amplitude for each frequency measurement. Additionally, it places uncertainty or error bar by taking the standard deviation of the residual. Note that this is not a standard way to place an error bar since it does not take the number of measurement points into consideration. According to the fit, the residuals were found to be usually a few counts which is much smaller than the amplitude of signals which was about 2000 counts. So it typically places 0.1% uncertainty after all.

The result is shown in the attached pdf. By the way the lower panel in the plot says, "residual", but it should read "(measured)/(model)" instead. It shows the measured transfer function together with the expected model transfer function for comparison. It is very obvious that the measurement suggests that our model is missing some high frequency pole. The model is merely made of the analog AA response which I have already measured and fitted. Adding some random pole, I could see an extra pole at around 10.7 kHz making the fitting much better. In fact, I sort of knew that there seemed to be a high frequency pole by some other measurements which I did not post. We probably need to add this high frequency pole in our calibration model.

We pulled both the power supply and the harmonic generator from the remote rack and tested them in the lab, together with spare supply and spare generator. We used a spare 9.1MHz source on the bench.

In the lab, regardless of the combinations (supply and generator), 45MHz thing was clean.

In the rack, we tested the original generator with spare and original supply, with 9.1MHz source coming from the distributor in the rack. In both of the cases there were huge 45+-2MHz-ish humps.

Evan reported in the past that using IFR at the rack didn't change the results, so we didn't bother to look at 9.1MHz source.

In all of these test cases both in the lab and rack, the only thing connected to the harmonic generator was power supply, 9.1MHz source and the network analyzer on 5x output. Everything else was terminated.

We also looked at the supply voltage in the lab. We opened the chassis and used clips to access gnd and +15V test points. Under the load, both of the units didn't show any huge high frequency noise. RMS voltage measured on the scope was about 1mV, which was dominated by some pickup (it got smaller when I hand held the chassis away from the 9.1MHz source). Fil also measured the spectrum and it was within the spec. And anyway, the harmonic generator didn't show any hump in the lab, so the power supply is pretty much exonerated.

Fil put the original units back in.