After making a few "OBSERVE.snap" files for a few SDF nodes during the DAY shift, I tried taking the nodes back to their "safe.snap" files, but some of the nodes ended up having differences!  Since we need to have ZERO differences to allow going to OBSERVATION Mode, I did what I could to make the differences as close to a "safe" state as possible.  For the "safe" files which had differences, I looked used whatever .snap file which got us as close to GREEN as possible. 

Here are the Nodes I worked on, the .snap file they are currently running, and a note about differences observed:

    1.    SUS-ITMX:  safe & GOOD!——> but now has nutsinee differences!
    2.    SUS-ITMY:  safe & GOOD!
    3.    SUS-ETMX *:  safe_150902_124725 & (1) Diff:  H1:SUS-ETMX_L2_DAMP_MODE1_GAIN ???, set point 0 & currently 50—>violin 1004.54Hz
    4.    ISI-ITMy:  safe_150902_124924 & GOOD!
    5.    SUS-ETMY:  safe_150902_151039 & GOOD!
    6.    HPIITMY **:  safe_150810_115155 & (1) Diff (H1:HPI-ITMY_OUTF_H2_TRAMP?? , set point 60 & currently 5)—accept
    7.    ISIBS:  safe_150902_151938 & GOOD!
    8.    HPIBS:  safe & GOOD!

* The SUS-ETMx difference was a gain change Nutsinee made.  She returned the gain to 0.

** The HPIITMY difference was a Time Ramp, so I ACCEPTED.

To damp ALL 8 frequencies of the ITMX first harmonics.

MODE1 damps 996.53

MODE7 damps 995.36

MODE8 damps 995.64

MODE9 damps 992.43, 992.79, 994.28, and 994.73

MODE10 damps 996.25

MODE1, 7, 8, 10 MUST stay on if MODE9 is on.

The damp settings have been added to the table.

One of the issues that has caused us a few locklosses over the last several days is a transient that happens when we engage a boost in DHARD P. 

The solution that has worked at least once today was to comment out the engaging of the boost in the state CARM 5 pm, and engage it latter in the sequence when the loop has higher gain(21133) .  However, this is the boost that has saved us from the DHARD pit instability in the refl trans state that plauged us durring ER7 (19243) and for a long time before that. We would like to engage it before transitioning the CARM control to the refl sensor to avoid the return of this instability.

I downloaded some data of transitions through the CARM 5 pm state.  In the attached screenshot the red traces are from locking attempts which failed either in this state or the next one, and the blue traces are from locks which survived. It seems as though this is a transient from engaging the first of the boosts, which had zero history and a 0.1 second ramp.  The impulse response of this filter is about 2 seconds long, and the DC gain is about a factor of 2. 

Evan and I set this filter to zero history, immediately, and also changed the 23Wboost in DHARD pit, from zero history 0.1 second ramp, to zero history immediately. 

We also uncommented line 1321 in the ISC_LOCK gaurdian, ezca.switch('ASC-DHARD_P', 'FM1', 'ON'), so that this filter will be engaged before the transition of CARM to the refl sensor. 

Cheryl will load the new filter and reload the ISC_LOCK gaurdian if we unlock, and knows how to undo the changes if necessary. 

Confirmed with a trend that the ambient temperatures have pushed the EX chiller into and past the low alarm levels.

The Major/Low alarm at 23:42UTC had a temperature of 34.99F.

The channel went back in the NO_ALARM state at 23:47UTC with a temperature reading of 35.45F.

I followed the alarm handler guidance and made a 10 day trend which shows that this has been happening for a while, so no notification/action needed.

J. Kissel, C. Gray, B. Weaver

Every once and a while while Betsy and Corey are "beating the streets" to find out why there are SDF differences between the settings definition file ("safe.snap") and the current value of the EPICs record for every front-end model, they ask me about differences in the CAL-CS model. Often times, including this time, there are hardware injection channels that show up, simply because the INJ infrastructure lives in the CAL-CS model. Over time, I've been changing these channels to NOT MONITORED, because they all appear to be EPICs records that are constantly updated with information about the last injection, or some recent GRB alert, etc. I attach a screenshot of this list as it currently stands.

For convenience I type the list here and give it the 'ol college try to explain them:
H1:CAL-INJ_EXTTRIG_ALERT_QUERY_TIME < a numeric GPS time that changes any time when TINJ (the hardware injection scheduling software) queries new external alert occurs
H1:CAL-INJ_EXTTRIG_ALERT_TIME       < a numeric GPS time that changes whenever there is a new external alert
H1:CAL-INJ_EXTTRIG_ALERT_SOURCE     < a string record from what telescope the external alert came (SWIFT, FERMI, etc.) that changes whenever there is a new alert
H1:CAL-INJ_EXTTRIG_ALERT_ID         < a unique alpha-numeric record for each record that changes whenever there is a new alert (e.g. E156669)
H1:CAL-INJ_EXTTRIG_ALERT_TYPE       < a string record indicating the claimed astrophysical source of the alert (e.g. GRB) that changes whenever there is a new alert
H1:CAL-INJ_TINJ_ENDED               < a numeric GPS time when the last hardware injecton ended
H1:CAL-INJ_OUTCOME                  < a numeric code labeling the level of success of the last hardware injection
H1:CAL-INJ_TINJ_START               < a numeric GPS time when the last hardware injection started
H1:CAL-INJ_TINJ_STATE               < a status vector of TINJ
H1:CAL-INJ_TINJ_TYPE                < a string indicating the type of last injection
As I understand it, all of the EPICs records are essentially readbacks, and therefore should not be monitored by the SDF system.

You can peruse T1400349 is you'd like more details on TINJ and the hardware injection infrastructure. T1500197 is the document for the external alert system.

WP 5455

A new frame writer, h1fw3, has been added to the DAQ system.  This frame writer is a disk clone of the frame writer that Livingston is running, with the system identity changed to LHO h1fw3.  The purpose of this frame writer is to see if there are differences in retransmission request rates between h1fw2 and h1fw3.  The h1fw3 is writing science, commissioning, and trend files to a local disk.  The raw minute files are not being written by h1fw3. 

If all tests are successful, the new frame writers will be installed in a permanent configuration on Tuesday Sept. 8.

9/2 DAY Shift:  15:00-23:00UTC (08:00-16:00PDT), all times posted in UTC

Summary:  This morning we had an unhealthy H1.  The first few hours were devoted to troubleshooting H1 & solutions found (see Kiwamu's alog).  H1 has been running for over 6hrs (but with fairly big glitches) & has been in/out of Observation Mode.  It looks like Hardware Injections have been running most of the shift.  PEM injections started around 1:20pm.

Had some issues saving .snap files from the SDF interface.  Have talked to Betsy & Jamie about it.  RED SDFs were GREEN-ed, and I will post an alog with notes on this.

Support:  Jenne handed off to Kiwamu & then Sheila/Betsy later joined.

Activity Log:

• H1 Investigations
• 15:53 Solar Panel work on grass out back (Filiberto, Richard, Gerardo)
• 17:37  H1 Taken to Observing Mode
• 17:55 Entrance to Elec Bay (CER) room
• 18:55 Pushed out of science mode when Betsy & I made SDF changes
• 19:54 Went back to Observing Mode
• 20:20 PEM injections begin by Robert/Anamaria in LVEA (taken to Commissioning mode)
  • 21:05 Now at EY
• 21:27 Gerardo investigating VAC issue at MY
• 22:28 Barker loaded filters for ETM PIs (because they have RED CFC on CDS Overview

I have created a simple H1IFO_RANGE.adl MEDM screen. This is mainly so users can get the channel name of Jonathan's IFO range epics channel. It is linked to the sitemap under the O-1 button.

To better manage the H1 IFO configuration changes during ER8 and into O1 I have committed all pending modifications of source files, filter files and safe.snap files into SVN.

Turns out I was the main culprit for model changes not being committed, the changes I made yesterday to odcmaster and pemcs had not been committed, nor my change some time ago to calex to add the irig-b channel to the science frame. I have committed these today.

The filter files for the new PI models were still static files in the chans directory. I moved them to the userapps/sus/h1/filterfiles area, made the symbolic link and committed them to SVN. The CFC bit was raised during this shuffle, so the DIAG_RESET button for both PI models were pressed to clear this bit.

All remaining local mods to filter and safe.snap have been committed with a generic "version as of 02sep2015 ER8" message applied.

J. Kissel, D. Tuyenbayv

After a quick look at the results from the UIM driver measurements taken for calibration purposes (see LHO aLOG 20846), and hearing similar whisperings of confusion from Darkhan in his detailed fitting analysis (progress continues, aLOG pending), I got the chance yesterday to quickly take a look myself to confirm my initial suspicions about the FAST I MON response to DAC drive measurements we took, and how they're "missing" the 85:300 zero:pole pair; see LHO aLOG 21127. I drove myself nuts for a half-a-day trying to reconcile the overall gain on the UL coil with what was expected to be sure *that* was right -- assuming it would provide clues if I couldn't reconcile it. I couldn't. 

I subsequently measured the respose of all UIM drivers of all QUADs and confirm that I was not insane. While the jury is still out on my sanity, at least I can confirm that
- The FAST I MON / TEST L EXC response for all four coils on all four of the Modified UIM Drivers are missing the expected 85:300 Hz zero:pole pair 
  EDIT: This is because the FAST I MONs are immune to this frequency response. See Below.
- This measurement techniue reports a factor ~2 less drive strength on the ETMY UL and LR channels, likely a busted monitor board.
- ITMY's UR and LR monitor board signals are total hosed.
I'll update the Integration Issue #9 with the later two bits of information.

EDIT: Why are the FAST I MONs Insensitive to the 85:300 [Hz] zero:pole pair? Turns out I just can't follow my own math. As it clearly states in My Notebook, you need to divide the current monitor by the output impedance -- i.e. whatever's in-between the voltage monitor pick-off and the coil current pick-off. Hence my equation I copied in LHO aLOG 21127,
                  R24_{MON}       1                   1     1       1      
DC calibration  = --------- x ---------- x G_{ADC} x --- x --- x ------- 
                  R25_{MON}   2 R5_{UIM}             E2O   CBG   G_{DAC} 
is wrong, and it should be
                  R24_{MON}       1                   1     1       1    
DC calibration  = --------- x ---------- x G_{ADC} x --- x --- x ------- 
                  R25_{MON}    2 Z_{OUT}             E2O   CBG   G_{DAC} 
where Z_{OUT} is formed by the whole R4, R5, and C12 ( = R27, R23, C26) network. This network is from where the 85:300 [Hz] zero:pole pair originates. Once I divided the transfer function with the right impedance -- voila! -- the 85:300 zero:pole pair appears.

The message again: The FAST I MONs are immune to the very frequency response we want to measure. So is every other monitor circuit, so we *have* to measure these drivers using analog electronics if we want to characterize this pole zero pair with any high precision than calculating what it should be from the component values. That being said, I don't think we'll time for this. In its stead, in case we don't find time, I checked the relevant resistor values inside spare modified UIM driver, and they we *at worst* 0.3% discrepant for all four channels on the spare. It's *very* regrettable that the monitor board appears to be giving discrepantly low DC transconductance, because that begets suspicion that the compenents are wrong, but I would be MUCH more inclined to blame the monitor board than the actual coil driver circuit given that the coil balancing gains are so similar, and using the stage hasn't brought up any issue.

Importance of this in the big picture: this re-opens the mystery of the frequency-dependent residual seen between naive model and measurment in the early results of the UIM actuation strength in LHO aLOGs 21015 and 21049. 

Stay tuned for Darkhan's results fitting the coil driver response poles and zeros that *are* there. After O1, we'll look into perfectly compensating all of these electronics, so we never have to think about this again.

The contractor has completed the 2nd phase of the installation which included the agent tank, piping and nozzle.

The remaining work involves the high voltage termination after which Fire Protection Specialists will return to test the system and turn it over to us.

(John, Gerardo)

Noticed the pressure along the Y-arm was rising, but there was no alarms of any kind, after investigating a bit, I determined that the pressure started rising at the Y-Mid station before any other place.
Went to Y-Mid and found the Ion pump controller tripped with the following error "SYSTEM ERROR SEE MANUAL ER7", still looking into this error, called John and he suggested to power cycle the controller, I did and the controller is now up and running.  The pressure is dropping.  We will keep and eye on this controller.
Attached is pressure trend for 1 day along the Y arm.

(Betsy, Corey, Jamie)

This morning I worked with Betsy (& Jamie over phone) on the SDF/Configuration Control (it actually knocked us out of Observation Mode at 18:55UTC!).  SDF is now being used to measure part of the configuration for H1, AND it must be in GREEN (no diffs) in order for an operator to go to Observation Mode.  In order to make SDF more all-encompassing, there was a thought to have SDF note the state of all channels when H1 was in Observation Mode.  (So, SDF would "light up like a Christmas Tree when in acquisition, but should be all GREEN when Guardian is complete. 

To test this out, we randomly picked ITMy.  We decided to Monitor ALL* of its channels while at NOMINAL_LOW_NOISE, and then (1) save this file as OBSERVE.snap & (2) Load this file into SDF. 

* We did decide to NOT Monitor a small list of channels which will change from lock to lock.  For a suspension, this would be the OPTICALIGN pit/yaw biases.

I've been waiting for H1 to break lock, so I could watch the new OBSERVE SDFs, to see if they return to a GREEN state when making it to NOMINAL_LOW_NOISE again, but we haven't dropped out of Science.  I might revert these SDFs to their safe.snaps since this is test configuration, and I wouldn't want to prevent Observation mode tonight.

At any rate, below are the SDFs I made OBSERVE.snaps for:

1. SUS-ITMX (Not monitored:  Optic pit/yaw)
2. SUS-ITMY (Not monitored:  Optic pit/yaw)
3. SUS-ETMX (Not monitored:  Optic pit/yaw)
4. ISI-ITMy (Not Monitored:  H1:ISI-ITMY_ST1_CPS_[*]_SETPOINT_NOW, which might be making tiny changes all the time.)

To back out of using the OBSERVE files & load safe, one can always:

1. go to the SDF nodes for the ones noted above,
2. open "SDF RESTORE SCREEN" (top center)
3. click "! SELECT REQUEST FILE" on RESTORE window
4. select & open "safe.snap" file
5. click "LOAD TABLE" on RESTORE window
6. This will return you to where we were at

To make these OBSERVE files will take us out of Science Mode (since "diffs" will be noted albeit briefly).  I might try making some more now since we are out of Science Mode for PEM Injections.  Otherwise, Betsy will be back Monday to also help out with this.

[JimW, Jenne, Evan, DanH]

Relocking after we took the IFO down for some Cal measurements about 7 hours ago has been pretty painful.  However, we're back up.  We were back up, before DHARD Pit kicked us out.

Our biggest problem was with some glitches that we think are in one of the mode cleaner mirrors.  They went away after a few hours though, so we aren't really clear on what the problem is.  We see them in MC2 MasterOut channels, so the glitches do exist in the digital system.  But, loops are closed everywhere, so it's tricky to tell where they originate.  While these glitches were present, we couldn't lock ALS for more than a few minutes at a time.  We would lose the lock before we were able to lock the DRMI, much less transition off of ALS.  The 2nd and 3rd attachments show quiet and noisy times, respectively.

Later, we lost lock a few times at Switch_to_QPDs.  Last month, Sheila and I kept adjusting the TRAMP on the reduction of the CARM offset that happens in that state, so I changed it again tonight.  I still don't have any particular understanding of why the speed at which we reduce the CARM offset matters so much.  For a while, we had been making it longer and longer (up to a Tramp of 60 sec), and then eventually reverted to the original 10 second ramp time.  Tonight, I put in the guardian that it should do this ramp over 30 seconds, and it's worked a few times in a row.

Next up was a set of weird PRCL glitches.  The attached lockloss screenshot shows them appearing in PRCL out, and the effect they have on the power recycling cavity's 9MHz buildup.  Some time around here, although I don't remember the exact order of operations (Evan, do you remember?) Evan reverted the 9MHz RF source from the IFR (which he installed during the calibration break earlier in the evening) back to the OCXO.  I don't know if we have a way to tell, other than someone's memory of the time, whether we were on the IFR or the OCXO at the time of this PRCL weirdness.  Anyhow, I think this only caused one lockloss, and it doesn't seem to be there right now. 

We have been seeing DHARD pitch oscillations during the CARM offset reduction for the past few days now.  Sheila and I were going to do some loop measurements last night to see why the loop was so unstable, but last night had its own troubles.  Anyhow, Evan commented out the turn-on of DHARD pitch FM1, "boostLTE".  This boost increases the gain below 1 Hz, but eats about 70 degrees of phase at 1 Hz.  With this commented out, we were finally able to get to DC Readout.  After we were sitting on Nominal Low Noise for a while, DHARD Pit started oscillating.  We saw it in the ASAIR camera, and in the ASC control signals.  My current hypothesis (without doing any measurements...) is that the IFO isn't stable enough for us to handle this semi-delicate loop before the rest of the ASC comes on.  So, I have added FM1 of DHARD Pit back to the guardian, but I put it at the very end of Engage_ASC_Part3 so that it comes on last.  Since it didn't ring up for more than ~30 minutes, I think delaying the boost turn-on should be okay.

The last sticking point didn't cause a lockloss, but it did prevent the guardian from getting to Nominal low noise on its own: The IMC guardian got stuck trying to turn on the ISS.  The ISS guardian has some logic that if the absolute value of the 1-second average of some number (PSL-ISS_SECONDLOOP_SIGNAL_OUT16) is above 2.5, don't turn on the ISS.  If this happens, it returns to the Locked state, but the ISC_Lock guardian is still requesting ISS_ON, so I think it should go back to trying to preparing the ISS, and rechecking this number.  Instead though, it just hangs at the Locked state.  I tried hand-requesting ISS_ON a few times, with the same result.  I don't know what this channel means, and I don't know how conservatively that threshold was set, but to try to get the lock to complete, I temporarily changed the threshold to be 3.5.  (After we got past this state, I put it back to 2.5 and reloaded the guardian, so next lock it will be back to the original threshold).  What is this signal (it's totally unclear from the screens, and I haven't opened up the model to see if I can figure it out)?  What are the consequences of increasing this threshold?  Why is the guardian getting stuck?  I think we need to re-examine this guardian logic. 

We pulled both the power supply and the harmonic generator from the remote rack and tested them in the lab, together with spare supply and spare generator. We used a spare 9.1MHz source on the bench.

In the lab, regardless of the combinations (supply and generator), 45MHz thing was clean.

In the rack, we tested the original generator with spare and original supply, with 9.1MHz source coming from the distributor in the rack. In both of the cases there were huge 45+-2MHz-ish humps.

Evan reported in the past that using IFR at the rack didn't change the results, so we didn't bother to look at 9.1MHz source.

In all of these test cases both in the lab and rack, the only thing connected to the harmonic generator was power supply, 9.1MHz source and the network analyzer on 5x output. Everything else was terminated.

We also looked at the supply voltage in the lab. We opened the chassis and used clips to access gnd and +15V test points. Under the load, both of the units didn't show any huge high frequency noise. RMS voltage measured on the scope was about 1mV, which was dominated by some pickup (it got smaller when I hand held the chassis away from the 9.1MHz source). Fil also measured the spectrum and it was within the spec. And anyway, the harmonic generator didn't show any hump in the lab, so the power supply is pretty much exonerated.

Fil put the original units back in.