While we were in commisioing mode, we had a strange non stationary spectrum, with large non stationary noise also appearing in MICH and PRCL.  I quickly looked at some triple witness sensors, since we've had issues with these recently, but didn't see anything obvious.  

The strange noise started a few minutes before 2:50:34 UTC Sept 10, seemed to get better for a few minutes and then got worse again around 2:57:21. 

We don't think that this was related to any comissioning we were doing.  Evan had temporarily lowered the DARM ugf and put a notch in, (which should cause anything like this).  He undid both changes when we saw the unusual noise, but the noise was unaffected by his change.  

We aren't going to investigate any further right now since the problem seems to have gone away.  

Last night, I moved the PRM coil driver switching to happen before the CARM offset reduction (21321 ) with the idea that this would make the locklosses it causes less painfull as they would happen earlier in the sequence.  We tried it this way once and things seemed fine, but then Travis later had a hard time locking.  Many of his locklosses were due to PRM coil driver switching.  This morning TJ and I moved this back to the DRMI on POP state (after CARM ofset reduction)  and we also had locklosses caused by it, although we were able to make it to full lock sometimes.  

This afternoon TJ and I looked into this a bit.  We looked at the amplitude of the glitches as seen by noisemon, and they are the same today as they were August 30th.  We also looked at how the PRM dirves respond to this glitch. Two weeks ago, we were nearly saturating PRM M3 each time we switched, in the past few days we have been nearly saturating M1 as well.  This is not suprising with the higher offloading gain intended to help us lock durring high ground motion (alog 21216)

I put an elliptic low pass at 10 Hz in PRM M1, this reduces the glitch in M1 by an order of magnitude.  We've survived the switching 3 times since we did this, and it is in the DRMI guardian.  In the two screenshots you can see an example of the glitch with switching this morning, and the second one shows the glitch with the new low pass engaged.  We are still in danger of saturating M3, so it would still be a good idea to look at tuning the delay if we get a chance.  

Log:

Times in UTC (PST)

• 1500 (800) Travis handing off the IFO not locked. He had issues for the past ~4:30 of his shift.
• 1557 (857) Nominal Low Noise. There was a problem with the PRM switching, commissioners moved it to an earlier state to save time, but this did not work.
• 1701 (1001) JimB to power cycle Trendwriter0
• 1727 (1027) Kyle to EX to unplug aux cart and then plug in ion pump
• 1740 (1040) Trendwriter0 id dead until further notice
• 1756 (1056) Dick G to optics lab
• 1812 (1112) Kyle back
• 2115 (1415) Evan to CER

Summery:

Commissioning or other activities for my entire shift. Not much to report from me.

Hugh and I have switched the SDFs of many of the SEI, SUS, the LSC, and ASC to compare current settings (OBSERVE.snap) to the NOMINAL LOW NOISE LOCK from around 2:30pm (local) today.  It then broke lock so we have to wait for the next lock stretches to finish - likely to be done tomorrow, so bear with us while we get this up and running.  Background:  we are redirecting to use SDF as the main settings configuration control monitor for O1, so we are now working quickly to do some smallish reconfiguring.  This involves setting the SDF such that it will monitor ~all settings of the IFO and will be all-GREEN when the IFO is in NOMINAL LOW NOISE.  This means it wiill have lots of reds after lockloss and during lock acquisition.  (We will work to reincorporate it's feature to be a troubleshooting tool soon.) 

For the next day or so, while in-flux, it may mean that there are some red SDF alarms that will not let you hit the INTENT button.  If this is the case, please feel free to call me and we'll work through the short diff list, or simply set any DIFFs to NOT MON and I will adjust them as needed tomorrow.

In the long run, the idea is that the SDF will monitor all settings, including guardian manipulated ones, except the few things which change between lock stretches (such as VIOLIN mode damping, HPI Setpoints, ISS REF CAV adjustments, and OPTICALIGN slider values).

One possible cause of the DAC overflows we've been seeing (the ETMY saturation warnings) is something at high frequency using up the range of the drive. We only record the last output to the ESD DAC at 2048 Hz, so we don't see anything happening above a kHz. It's possible some high frequency line or feature bumps up against 2^17 counts and that makes the DAC overflow, which then causes a much larger glitch - once there's an overflow the whole thing will go crazy.

To try to investigate this, the simplest thing to do is to record a channel like SUS-ETMY_L3_MASTER_OUT_LL_DQ at the highest rate possible, or something equivalent that monitors the actual signal sent to the ESD DAC. Once an ETMY overflow happens, save the data for several seconds around the time somewhere that people offsite can access it, preferably in a common format like ASCII or frames. Some time without an overflow would be good for comparison. Then we can see if there's something using up the drive range, or if it's just something sudden from somewhere else in the DARM loop. A few instances of overflows would be nice.

If this is not possible, there's another tack we can take but it's more involved, so I'd prefer this.

Similar to what I did yesterday for HEPI re 21309 I made OBSERVE.snap files for all the ISIs.

However, I came upon an issue for the BSC ISIs where toggling the MONITOR SELECT choice button to ALL left the SDF with differences.

These come from the following: When an ISI or HEPI Isolates, it observes and averages the free hanging position for 5 seconds and puts this mean into the SETPOINT_NOW.  This makes the residual error of the loop zero when it is turned on.  After the loop is completely on, the Guardian puts the Reference Location (e.g.: H1:ISI-BS_ST1_CPS_X_TARGET) into the SETPOINT_NOW record causing the Isolation loop to servo to that same position from platform trip to platform trip.  That is for some but not all platforms/dofs.  All HEPIs and all HAM ISIs are resorting the TARGET.  No BSC ISI is restoring the Target location.

So even though I just took a new OBSERVE.snap, many of the BSCs showed diffs out at e-10 to -15 in the SDF.  I'm not sure why that was variable but it brought the realization forward that each time a BSC ISI tripped, the new SETPOINT_NOW would be different (as it would for all the HEPIs and HAM ISI,) and would not be replaced by the TARGET Location at the end of Isolation (as it would be for the HEPIs and HAM ISIs.)

So for the BSC ISIs, after getting the OBSERVE.snap loaded into SDF as done for the HEPIs and HAM ISIs, the NOT MONITORED channels were all set to monitor and then the SETPOINT_NOW channels were set back to NOT MONITORED.  Given this, the BSC ISIs will not be pink and the MONITOR SELECT button will be set to MASK, rather than ALL like the HEPIs and HAM ISIs.  It may be that I will do the same for all the HEPIs and HAM ISIs if that is best for consistency, TBD.

It seems like shifting the pole from 120Hz to 1.2Hz (alog 21300) reduced the beam jitter greatly only at 60 and 180 Hz. The broad jitter noise floor might have been somewhat reduced between 10 and 100Hz, but I see no change at 300Hz. In the attached, references are before, current traces are after.

Robert reported that the coupling to the chiller water flow dominates the periscope motion at around 300Hz (alog 21273).

Laser Status:
SysStat is good
Front End power is 33.02W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED

PMC:
It has been locked 8.0 days, 1.0 hr 19.0 minutes (should be days/weeks)
Reflected power is 2.097Watts and PowerSum = 26.62Watts.

FSS:
It has been locked for 0.0 days 1.0 h and 10.0 min (should be days/weeks)
TPD[V] = 1.498V (min 0.9V)

ISS:
The diffracted power is around 7.045% (should be 5-9%)
Last saturation event was 0.0 days 1.0 hours and 10.0 minutes ago (should be days/weeks)
 

In prep for turning ~all settings configuration control over to SDF, I have turned off the Set Point Monitoring (SPM) feature on the SUS guardians that were in false alarm (yellow).  We will not be using the SPM feature as a requirement for O1, although any ones that go into alarm which are stll enabled may be used as a good troubleshooting tool.  (Most that are enabled have been green and running for many months now, so can't hurt to stay on.)

Running pump cart in parallel near BSC9 until ion pump reaches operating temperature 

(TJ writing as Travis)

Reminder to Operators: We are NOT adding a second stage of whitening anymore. We are calibrrated for only one stage, and a rare case of zero.

The SSD RAID attached to h1tw0 (and h1nds0) has failed.  Until the problem can be diagnosed and fixed, minute trend data won't be available from h1nds0.

late entry for yesterday's CDS maintenance

DAQ frame writer install

Dave, Jim:

The second fast frame writer was connected to the LDAS Sataboy-raid system. All frame writers were then renamed, details in Jim's alog

Adding Duotone Channels to the Science Frame

Keita, Dave:

The h1omc, h1calex and h1caley models were modified to all DUOTONE timing signals from all three buildings to the science frame at 16kHz. These signals are being readout by a filtermodule, to capture the raw signal before any filtering/gain/offset could be applied, we are using the IN1 channel.

H1CALEX.ini:[H1:CAL-PCALX_FPGA_DTONE_IN1_DQ]

H1CALEY.ini:[H1:CAL-PCALY_FPGA_DTONE_IN1_DQ]

H1OMC.ini:[H1:OMC-FPGA_DTONE_IN1_DQ]

Loading all the filters

Sheila, Dave:

SUS PRM and SRM had partially loaded filters, we performed a full load of these systems.

Restarted External Alerting System

Dave:

I restarted the external alert system which had stopped between 6pm and 7pm PDT Monday. I'm looking into a mechanism to restart this system.

J. Kissel, K. Izumi

We haven't gotten these two measurements since the tail end of Calibration week, and we want to confirm that our model (that's under development) is still valid after 1.5 weeks, so we've taken new DARM OLG TFs and PCAL to DARM TFs. We'll analyze these in the next few days and confirm what we can determine from the calibration lines.

Data lives here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/
DARMOLGTFs/2015-09-08_H1_DARM_OLGTF_7to1200Hz.xml
PCAL/2015-09-08_PCALY2DARMTF_7to1200Hz.xml

For the record, this pair of measurements takes almost exactly 1 hour. However, to knock down the measurement time to that one hour, one needs to run the DARMOLGTF first, and start the PCAL TF once the DARM OLG TF hits the frequency points around 15 to 10 [Hz]. 

Note, calibration lines were turned OFF during these measurements. They're now back ON.

Collected OBSERVE.snap files for use in full observing configuration monitoring.

With the SDF happy (green) and Guardian Nominal (Robust Isolated) and green, I made an OBSERVE.snap file with SDF_SAVE screen

choosing EPICS DB TO FILE  & SAVE AS == OBSERVE

This saves the current values of all switches and maintains the monitor/not monitor bit into the target area, e.g.   /opt/rtcds/lho/h1/target/h1hpietmy/h1hpietmyepics/burt as OBSERVE.snap

This file is then copied to the svn: /opt/rtcds/userapps/release/hpi/h1/burtfiles as h1hpietmy_OBSERVE.snap and added/commited as needed.

The OBSERVE.snap in the target area is now deleted and a soft link is created for OBSERVE.snap to point to the svn copy:

ln -s /opt/rtcds/userapps/release/hpi/h1/burtfiles/h1hpietmy_OBSERVE.snap OBSERVE.snap

Finally, the SDF_RESTORE screen is used to select the OBSERVE.snap softlink and loaded with the LOAD TABLE button.

Now, for the HEPIs for example, the not monitored channels dealt with by the guardian will be a different value from the safe.snap but, the not monitored channels are still not monitored so the SDF remains green and happy.  And if the HEPI platform trips, it will still be happy and green because, the not monitored channels are still not monitored.

What's the use of all this you say?  Okay, I say, go to the SDF_TABLE screen and switch the MONITOR SELECT choice to ALL (vs MASK.)  Now, the not monitored channel bit flag is ignored and all records are monitored and differences (ISO filters when the platform is tripped for example) will show in the DIFF list until guardian has the platform back to nominal.

Notice too that the SDF_OVERVIEW has the pink light indicating monitor ALL is set.  This should stay this way unless Guardian is having trouble reisolating the platform and then the operator may want to reenable the bit mask to make more evident any switches that guardian isn't touching more apparent.

Evan, Sheila, Jeff B Ed

Earlier today we were knocked out of lock by a 5.2 in New Zealand, the kind of thing that we would like to be able to ride out.  The lockloss plot is attached, we saturated M3 of the SRM before the lockloss and PRM M3 was also close to saturating.  

While LLO was down, we spent a little time on the offloading, basically the changes described in alog 21084.  This offloading scheme worked fine in full lock for PRM, however we ran into trouble using it durring the acquisiton sequence.  Twice we lost lock on the transition to DRMI, and twice we lost lock when the PRM coil state swtiched in DRMI on POP.  Hpwever, we can acquire lock with the new filter in the top stage of PRM and SRM, but the old low gain (-0.02).  We've been able to turn the gain up by a factor of 2 in full lock twice, so I've left the guardian so that it will turn of the gain in M2 (before the intergrator) in the noise tunings step. 

If anyone decides they need to undo this change overnight, they can comment out lines 344-347 and  2508-2514 of the ISC_LOCK guardian. 

Before we started this, the PRM top mass damping was using 10000 cnts rms at frequencies above a few hundred Hz, because of problems in the OSEMS (alog 21060 ).  Evan put some low passes at 200 Hz in RT and SD OSEMINF which reduces this to 2000 cnts rms. Jeff B accepted this change in SDF.  

 The second attached screenshot shows the PRM drives, the references are in the minutes before the earthquake dropped us out of lock.  The red and blue curves show the current drives, with the high frequency reduction in M1 due to Evan's low pass, and the new offloading on.  The last attached screenshot shows SRM drives with the new offloading.  

• Coil drivers all in low noise state.  Now all suspensions will be in their low noise state once the guardian has completed.
• We tried turning off ITM opLev damping, which eventually caused an alignment instability. 
• SR3 OpLev damping has been off all day, but we seem to be stable at high power without it. 
• We tried reducing the ETMY ESD bias by 33%, and increased the gain in drivealign.  
• Hang did some A2L tuning, and saw that the coefficients had changed. 
• Matt and I looked at Kiwamu's calibration line, and used an rms monitor of it to try to adjust the OMC QPD offsets to maximise the optical gain.  We changed the offsets slightly, but we were mostly only able to decrease the optical gain.  We noticed that the optical gain is moduated by some low frequency (microseism).  The seismic noise has been verry low.  
• There is some MICH coherence between 40-80Hz, we can revisit MICH FF tomorrow. 
• We measured CHARD PIT A2L, we would project that this is a factor of 2 below DARM at 20 Hz, and a factor of 5 below at 50 Hz.  This was with the low pass off, before Stefan's work.
• PI monitor is running for the Y arm.
• Evan updated some calibration, alog on the way tomorow. 
• Evan did not hit load on all the unmodified filters in various models, since the IFO is locked.  
• Intent bit is undisturbed.

None of these things changed the DARM noise.  (just the calibration)