Although the error reporting is not detailed enough yet on the production server to provide the channel name and value that caused an error upon attempting to insert it into the MySQL database, it is on the test system. The following channels and values caused errors on the test system:

Jul 11 16:59:40 H1:OMC-READOUT_ERR_GAIN: -nan
Aug  4 07:54:43 H1:SYS-MOTION_C_PICO_F_MOTOR_1_NAME: 'xF3x01'
Aug  8 12:25:47 H1:OMC-READOUT_ERR_GAIN: -nan

LVEA: Laser Hazard
IFO: Locked
Observation Bit: Undisturbed    

All Times in UTC (PT)

15:00 (08:00) Take over from Ed
15:00 (08:00) IFO locked at NOMINAL_LOW_NOISE, 68Mpc
15:38 (08:38) Switch to commissioning/calibration mode for commissioners
16:15 (09:15) Alarm on Instrument air MR-PT199 – Left message for Kyle & spoke to Bubba
16:25 (09:25) Bubba – Check into alarm with PT0199
16:30 (09:30) Lockloss – Due to commissioning – Put IOF in DOWN per commissioner request
17:19 (10:19) Kyle – Going to X2-8 near End-X for vacuum work
17:59 (10:59) Filiberto – Going to Mid-Y to get parts for PEM work
18:00 (11:00) IFO locked at NOMINAL_LOW_NOISE, 65Mpc
18:17 (11:17) Filiberto – Back from Mid-Y
19:26 (12:26) Fire Protection Specialist – On site working on alarm system in VPW
19:32 (12:32) Filiberto – Going into CER looking for tools
19:38 (12:38) Kyle – Leaving X2-8 going to Y2-8 on the Y-Arm
21:44 (14:44) Lockloss – Commissioners looking into cause
22:10 (15:10) IFO Locked at NOMINAL_LOW_NOISE, 65Mps 
22:19 (15:19) Lockloss – Calibration activities
22:32 (15:32) Lockloss – Possibly due to Earthquake

 
Shift Summary & Observations:

   Took over from Ed at 15:00 (08:00). IFO locked with good range. LLO down. 
   Commissioners have started working on calibration.
   IFO was stable during the day. After first lock loss the IFO relocked at NOMINAL_LOW_NOISE in 34 minutes. 
   21:56 (14:56) 5.2 Mag EQ at N. Mid-Atlantic Ridge R-Wave arrival ~22:28 (15:28)  

We are testing out a DMT to EPICS bridge that is able to translate DMT data to EPICS data real time.

The channel names are subject to change (comments should be directed to Dave Barker, not the alog).  Currently they are not being recorded in the frames.  There should be a discussion regarding whether or not it is a good idea to put another copy of the range in the frame with a different name and a slight time offset due to being captured from EPICS.

Current channels:

• H1:CDS-SENSMON_CAL_SNSW_EFFECTIVE_RANGE_MPC
  • This is the inspiral range as displayed in the DMT viewer in the LHO control room.
• H1:CDS-SENSMON_CAL_SNSW_EFFECTIVE_RANGE_MPC_GPS
  • The GPS time associated with the data in the range channel.

Please remember that the range is a really a minute trend, and that it is visible at some point within a minute of it being generated, see the _GPS channel for the time it refers to.

There are a few other approaches we can take to getting DMT data into the control room faster than the frames appear, so if this does not work out there are some other avenues open to exploration.

Keita updated the ASC model to come inline with LLO channels they are using.  The OM3_PIT1 & YAW1 FM channels changed to just OM3_PIT & YAW.

These changes caused the SDF overview to go yellow for ASC as there were now channels not found and channels not initialized.

Removed these lines from the safe.snap to clear the channels not found:

+H1:ASC-OM3_PIT_GAIN 1 0.000000000000000e+00 1
+H1:ASC-OM3_PIT_LIMIT 1 0.000000000000000e+00 1
+H1:ASC-OM3_PIT_OFFSET 1 0.000000000000000e+00 1
+H1:ASC-OM3_PIT_SW1S 1 0.000000000000000e+00 1
+H1:ASC-OM3_PIT_SW2S 1 5.120000000000000e+02 1
+H1:ASC-OM3_PIT_TRAMP 1 0.000000000000000e+00 1
+H1:ASC-OM3_YAW_GAIN 1 0.000000000000000e+00 1
+H1:ASC-OM3_YAW_LIMIT 1 0.000000000000000e+00 1
+H1:ASC-OM3_YAW_OFFSET 1 0.000000000000000e+00 1
+H1:ASC-OM3_YAW_SW1S 1 0.000000000000000e+00 1
+H1:ASC-OM3_YAW_SW2S 1 5.120000000000000e+02 1
+H1:ASC-OM3_YAW_TRAMP 1 0.000000000000000e+00 1
 

The new channels are not used at LHO but the path came up with the gain of 1 and the IN and OUT of the FMs engaged.  There is no medm for these FMs so I set the gains and filters to 0 and OFF with CAPUT and EZCASWITCH.  Will this be an ODC issue?

See the attachment for the list of new channels and their current values.  Snap committed.

ECR 1500312, WP 5455

A new frame writer, h1fw2, has been added to the DAQ system for H1.  It is currently writing science and commissioning frames to it's local file system to allow comparison with existing frame writers.  

This frame writer is an Ubuntu 12.04 based system, with a daqd compiled from advLigoRTS/trunk rev 4057. The computer is a new dual 10 core CPU computer with 132G of RAM.  The ultimate goal is to see if this configuration can be used to replace the existing frame writers.

At this point, h1fw2 should have no impact on the DAQ system unless there is an effect on the data concentrator (h1dc0). We don't anticipate any effect, this configuration has been running at Livingston for at least a week.

After some email communication with Keith Thorne and Elli, I've added limits at 32,500 to the channels L1:TCS-ITMX_CO2_CHILLER_OUT_GAIN_OUT and its y-arm sibling.  This has also been done at LLO.

These channels have a static output on them, and are used to set the chiller temperature for the TCS CO2 lasers.  The value is not changed, and will not be at all affected by the limit being put in place. 

ER8 Day 9. Maintenance Morning

model restarts logged for Tue 25/Aug/2015
2015_08_25 11:06 h1oaf
2015_08_25 11:17 h1oaf
2015_08_25 11:19 h1odcmaster
2015_08_25 11:20 h1lsc
2015_08_25 11:22 h1asc
2015_08_25 11:22 h1calex
2015_08_25 11:26 h1broadcast0
2015_08_25 11:26 h1dc0
2015_08_25 11:26 h1fw0
2015_08_25 11:26 h1fw1
2015_08_25 11:26 h1nds0
2015_08_25 11:26 h1nds1
2015_08_25 23:17 h1susprm
2015_08_25 23:22 h1lsc
2015_08_25 23:24 h1asc
2015_08_25 23:28 h1suspr2

No unexpected restarts. Model changes followed by supporting DAQ restart. Later model restarts in attempt to fix n+0.5 Hz noise comb.

ALL TIMES IN UTC

Post-Maintenance: IFO is actively locking; caveat-> excess noise that wasn’t present prior to maintenance (1Hz comb and some high freq). Calibration going on. Wind calm. Seismic calm. Patrick on Evening duty.

Activity Log: Not much activity. Folks here from the day eventually left.

• 7:00 Sheila, Evan, Dan and Jenne on site doing their thing(s)

• 7:29 Got SDF cleared out except for CALCS (will remain Red with diffs)

• 7:29 Evan running excitations on H1LSC is the only thing left that is keeping H1 from Observing mode at this time

• 8:27 Added OMC whitening. Not able to set the Intent Bit

• 8:56 Evan and Sheila leaving site.

• 12:09 ETMY saturation

• 12:15 LLO dropped out of science mode

• 12:28 ETMY saturation

• 12:59 ETMY saturation (2)

• 13:45 ETMY saturation

• 13:56 Took IFO to Commissioning mode for Richard to investigate the 1Hz comb in PRCL

• 13:57 ETMY saturation (3)

• 14:04 Set OIB/OOM back to Undisturbed/Observing. Noise in PRCL successfully removed

LOCK LOG:

• 7:00 Locked at NOMINAL_LOW_NOISE

• 8:43 OIB/OOM set after Evan figured out how to clear the ODC Observation Ready bit.

Summary:

• Uneventful night. IFO locked for ~8 hours.

• ETMY saturations started to occur midway through the shift. The first small one didn’t seem to be related to typical ETMY saturations.

• Took IFO out of science mode momentarily for noise maintenance in PRCL which was remedied by Richard McCarthy.

• Handing off to Jeff B

Set OIB/OOM to Undisturbed/Observing after noise successfully removed from PRCL. 

[Sheila, Dan, Evan, Jenne, others]

All afternoon (basically, since we started relocking after maintenence day) we have seen a 1 Hz comb (offset by 0.5 Hz) in the DARM spectrum, as well as in many, many other places.  This seems like the same thing as the final bullet point that was reported in aLog 20790 a few days ago, but with much taller peaks.

Much of the day was spent looking into this, by several people.  The lines are very clear in all of the length error signals, as well as the ASC error signals. 

One thing that we tried, just to exonerate it, was swap the PRC length actuation from the PRM to PR2, and then turn off the PRM M3 coil driver.  As expected, PR2 affects the cavity length twice as much as the PRM, so we needed a gain of 0.5 in the M3 LOCK filter bank for PR2, rather than the 1 that PRM has.  We were able to turn off the PRM M3 coil driver, but saw no change in the length signals' spectra.  We put the actuation back on PRM, but when we tried to turn off the PR2 coil driver, PR2 tripped and we lost lock.  So, the new PRM coil driver is certainly not to blame for this new noise.  (PR2 was untripped, and its coil driver turned back on)  Also, we haven't seen the glitches due to the old PRM coil driver at all today, so hopefully they're gone for good, and not just hidden.

At Richard's suggestion, we tried restarting the models that were restarted earlier today in case it is some kind of synchronization problem.  We restarted h1prm, h1pr2, h1lsc, h1asc.  This didn't change anything - the lines are still there loud and clear after relocking. 

The 1 Hz comb is still present, and we are running low on ideas for what it could be, and what to do about it.  DetChar friends, and anyone with ideas, please let us know if you have a suggestion of where to look.

--------------

I don't know if this has the same cause as the 1Hz comb, but there are also some unusually large lines at ~838 Hz and ~855 Hz.  These lines are a few Hz wide, and are coherent with magnetometer signals.  But, the magnetometers have seen these lines at the same (or larger) amplitude, in the past, without seeing the lines in DARM.  Why would the coupling from magnetic fields have changed so drastically??

Sheila, Patrick, Jenne, Eric Thrane on the phone.  

Eric wanted to do an injection, and since his code requires the intent bit to be set, Patrick attempted to set it.  This however proved impossible.  This is probably related to the work described in 20870.  According to the CDS overview, there are no excitations on except those from calibration, which will need to be on while the intent bit is set.  (screenshots attached of the CDS overview and some ODC screens). 

Side note (sheila's opinion only):

I found the ODC screens confusing to navigate, and it is very unclear to me which bits go into calculating other bits to find out what the problem is.  It seems to me like it is a bad idea for us to make ourselves dependent on ODC to set the intent bit, since people on site don't have control over it.  

Following up on Evan's investigation into the glitches from LL coil on the PRM M3.   I tried to monitor the output with a scope attached.  I was not looking for very big signals as the the noisemon indicated roughly 30mV signal and it has some gain associated with it. So I connected to scope probes to the LL coil drive signal and looked at it on a scope with a trigger set fairly low.(10s of milivolts).  I also looked at UL just for comparison.   I was able to get the signal to trigger on some events  on LL that were not seen at as high a level on UL.  This was with the DAC both connected and disconnected with small offset (20 counts) applied and a large offset (20,000 counts) applied.  After a couple of hours of investigation I decided there were some glitches present and seemed to be from the coil driver not the DAC.  I also looked at the DAC input to the coil driver and did not see the glitches.  In the end I swapped out the coil driver.  I am not 100% certain this will fix the problem as it was difficult to verify the glitches were present at all times.

Since LLO had been down in this afternoon, I took this oportunity to renew the dtt templates that are for measuring the transfer function of each stage of ETMY. I tuned the envelop parameters while the interferometer was locked at 23 W in the NOMINAL_LOW_NOISE state. The attached are the resultant templates. The frequency range is newly adjusted to [10 200] Hz with 21 frequency data points as planned. Each one takes several minutes to complete the measurement. According to the results I got, we can get a quite good coherence for all the relevant stages (i.e. L1, L2 and L3 stages) at all the frequency points with a coherence of more than 0.99. On the other hand, the template for measuring the DARM closed loop in this particular frequency band may need a bit more tuning because the coherence was not as great as the ones for the ETMY transfer functions.