Last night we saw a familiar bit of excess noise return -- a double-hump feature between 500Hz and 1400Hz, see first plot.  This noise had been intermittently observed up until March 3 (see Gabriele's discussion here) -- after this we were successful in closing the corner station ASC loops in low noise, and we thought we had driven it away.  But last night it was back.

The noise decays over a timescale of tens of minutes, which makes us think of a slow alignment loop that drags the IFO towards an operating point where the noise that is causing the bump doesn't couple to DARM.  Also the intermittent appearance of the noise makes us think that the coupling depends on the initial alignment, or at least something that changes from lock to lock.

The only alignment loops that were changing on the timescale of the noise last night were the YAW loops in the IMC.  The second plot attached shows a bandpassed trend of DARM_IN1 (top) and a trend of IMC-DOF_3_YAW, for the two hours as the noise disappears.

The third and fourth plots are a two-hour trends of other ASC signals (same timescale as the second plot).  The corner station ASC (third plot) and the ETM loops (fourth plot) arrive at their preferred output values fairly quickly, much faster than the decay rate of the noise.

Ryan Fisher, TJ Massinger:

We added an ODC channel to the h1omc.mdl, requiring changes to the library parts: omclsc.mdl and omc.mdl

We added an ODC channel to the h1asc.mdl, requiring changes to ASC_MASTER.mdl
  (With Kiwamu and Stefan)  We also added a matrix to the ASC_MASTER.mdl to allow for selection between AS_A and AS_B for the AS45Q PIT signal used for test mass damping. The output matrix is called OUTMATRIX_TESTMASS_DAMP.

We changed the configuration of the LSC ODC to account for the LSC/OMC model restructuring from alog 16893.  These changes were made to the lsc.mdl common library part. Channels that no longer exist in the LSC model were terminated inside of the ODC block and the bits were set to default to 1. We also added the ODC_CHANNEL_OUT to the annotation block to automatically record the ODC channel to the frames.

We made several changes to h1odcmaster (with Duncan Macleod):
  All SIM (simulated) inputs were removed as they are no longer needed for testing.
  The IPC links for ASC, OMC and LSC ODC channels were added.
  The RFM links from h1iscex/ey for the X and Y ODC channels were added.
  cdsEzCaRead links were used to read in Guardian states from ISC, IMC and ODC.  
  The model was converted to using a common library part for the ODC MASTER channel calcuation.  This library part includes the use of the new cdsEpicsOutLong parts to record the EPICs values as 32 bit unsigned integers.

Duncan M. updated the ODC MASTER MEDM screens to agree with the changes made to h1odcmaster.

Cleared HAM2, ITMX, BS, ITMY, ETMY

Scott L. Ed P. Chris S.

Cleaned 57 meters of tube towards X-1-5 double doors. Problems with the generator slowed the start this morning, John and I will investigate more tomorrow morning. Busy maintenance day and 3rd IFO duties prevented us from looking into the generator issues.  Fortunately we are at the only location that we have electrical power in the X arm enclosure and were able to relocate the extension cords to continue cleaning. Continuous monitoring of the beam tube pressures by the control room.  

Beam tube cleaning scheduled from 7:30 - 4

07:18 Cris into LVEA
07:30 Karen into LVEA
07:30 Peter into H1 PSL enclosure
~8:20 Corey to squeezer bay for 3IFO work
08:26 Sudarshan to end Y for PCAL work
08:57 Bubba and Jodi to move 3IFO parts from corner station to mid stations
08:58 Jim W. running measurements at HAM6
09:01 Filiberto to Y arm spool to remove electronics from test stand
09:05 Hugh and TJ cycling HEPI pumps
09:07 Richard to end stations
09:08 Rick, Ed and Jeff B. to H1 diode room to reset watchdogs
09:08 Betsy to LVEA west bay for 3IFO work
09:13 Daniel changing Beckhoff code
09:20 Peter transitioned LVEA to laser safe
09:21 Jim B. to LVEA to help Filiberto
09:27 Travis to LVEA west bay for 3IFO work
09:34 Jeff B. and Andres moved dust monitor in diode room off floor
09:39 Hugh and TJ done cycling HEPI pumps
09:39 Jamie restarting guardian machine
09:46 TJ to end stations to restock cables
09:46 Jim B. back
09:47 Peter K. to H1 enclosure
09:47 Rick to end Y
09:49 Elli to end stations to work on HWS cameras
09:53 Hugh to HAM1 to read HEPI load cells
10:09 Dave restarting H1 ODC master
10:10 Hugh done at HAM1
10:14 Peter and Matt done at H1 PSL
10:15 Betsy and Travis done
10:17 Keita to squeezer bay to talk with Corey
10:18 Travis and Betsy starting TFs on SRM, SR2 and SR3
10:38 Richard to squeezer bay to talk to Corey
11:03 Karen to end Y, Cris to end X
11:15 Gary and Jodi at mid X
11:27 Dave restarting end X ISC model
11:38 LN2 truck went through gate
11:51 Mayflower Metal delivering metal recycling container
11:57 Informed by Kyle that pumps are running on HAM1, HAM2 and BSC3
12:02 Peter transitioning the LVEA to laser hazard
12:12 Peter done transitioning the LVEA to laser hazard
12:20 ASC model changes done, Ryan starting on OMC, LSC, ODC master
12:24 Beckhoff end Y PLC3 EPICS invalid, Daniel restarted IOC
12:32 Karen leaving end Y
12:53 Praxair truck is ~ 15 minutes away
12:42 Corey to ISCT1 to take pictures
12:44 Jeff K. recompiling, restarting H1 OMC model for Ryan
12:52 Corey done at ISCT1
13:11 Cristina driving forklift from LSB to warehouse and staging building
13:11 Filiberto to mid Y
13:13 Rick and Sudarshan to end Y to work on PCAL
13:39 Jeff K. and Ryan resuming work on model changes for ODC
13:39 Betsy to LVEA
13:45 TJ to LVEA to hang cables in racks
13:48 Corey to squeezer bay 3IFO
13:48 Greg to LVEA to put parts on racks by HAM6 and look for filters to reuse from eligo chillers
14:10 Hugh taking HAM1 HEPI down to take safe.snap
14:14 Filiberto back from mid Y
14:19 Peter and Ed to H2 PSL enclosure to work on Livingston PMC
14:29 Travis and Betsy done running TFs
14:37 TJ done
14:57 Rick and Sudarshan done, Dave restarting models at end Y
15:07 DAQ restart
15:56 Corey out of LVEA

In order to expedite their removal, I am letting them run 24/7 until not needed (rest of this week?)

J. Kissel

I've compiled, installed, and restarted the h1calcs front-end code to absorb the changes described in LHO aLOG 17114 this morning. Most of the changes require brand-new custom MEDM screens (yaaaayyyy), so I'll post more of how the infrastructure has been filled in once I have some screens to show. For now, I've confirmed that the DARM calibration stuff is still ON and the input for the slow optical gain correction is turned OFF, and NOT changing the DARM calibration.

A couple of EtherCAT software problems were address today:

• Updated the "wiring" documentation for end station TCS.
• Wired the camera and fiber enable switches for the HWS at the end stations.
• TEC: Added save/restore and fixed an issue where the integrator gets reset, when it reaches its maximum output. Fig 1 and 2 show the step response for the old and new TEC controller, respectively.
• Also added minimum values for the green power and conversion efficiency, so that it now produces an error when disabled.
• Added a preset option for the VCOs with a PLL (ALS DIFF and COMM). Fig 3 shows the updated medm screen with the preset option in the right upper corner. No - normal operation; Low - Set VCO to lowest frequency, High - Set the VCO to the highest frequency, Mid - Set the VCO to mid frequency. Both ALS DIFF and COMM VCO should be parked at their Low preset when in full interferometer lock. See alog 17089 for details.
• Fixed an issue with the ALS DIFF VCO which was set up with two FDDs in software, even so we only use one. 

The h1asc awgtpman is a special version which will allow the full 64 test points to be set.  In order to do this, the awgtpman_h1asc.cmd script (generated by RCG) was replaced with a hand-edited version which starts up awgtpman-2.9-asc-special (instead of awgtpman) and also includes a -A option to the command line.

If the h1asc model is rebuilt, this will be undone by the RCG. Coordinate model rebuilding and restarting with Jim Batch in order to ensure the modified awgtpman is running.  

This is a temporary solution to allow commissioning to continue where more testpoints are needed.  We will decide on a complete solution for the next RCG release.

The two plots show the vacuum gauges at either end of the X1 module for the past 100 days.

The features shown are due to large gate valve closures at the corner station. PT144 is next to the corner station while PT343 is at the Mid station.

I removed the locations that are not currently used. This cleared all the invalid alarms. All that remain are the calibration errors for the dust monitors in the optics lab and bake out room.

J. Kissel, B. Weaver, J. Rollins

We (Betsy and I) had begun returning all the corner station SUS to ALIGNED while Jamie was restarting all of the Guardian code. After the dust settled, we noticed that ITMY and PRM were stalled in their SAFE state. After a little bit of poking around in the guardian logs, we discovered that upon initialization, LSC_CONFIGs manager took possession of SUS_ITMY and SUS_PRM as expected. However, a minute or so after, when the IFO_ALIGN manager initialized, it superseded control. Neither the LSC_CONFIGS nor IFO_ALIGN managers' initialization state is coded enough to acknowledge that its subordinate has stalled in SAFE and to issue its commands, so the SUS remained stalled in SAFE. 

This *didn't* happen on any of the other SUS because the SUS were already in the ALIGNED state -- the state that IFO_ALIGN (who had gained management this time) expects them to be in -- so no action was taken. Betsy and I had not yet gotten to requesting ALIGNED of PRM and ITMY. Therefore the nodes went down in the SAFE condition, restarted, initialized, found the SUS in the SAFE state, jumped to SAFE, and stalled because their manager didn't tell them to continue up to the requested state of ALIGNED.

In summary, there are several flaws that caused this issue:
- PRM and ITMY (and probably other SUS) are doubly managed by LSC_CONFIGS and IFO_ALIGN.
- LSC_CONFIGS lost management of PRM and ITMY, and didn't acknowledge it or throw a warning. Nor did LSC_CONFIGS notice the stall and tell the SUS to keep going.
- IFO_ALIGN doesn't know how to handle a stalled subordinate node; again, upon initialization, it doesn't acknowledge the node is stalled, throw a warning, or do anything about.
- Globally, the ISC managers were not properly initialized to send out the proper commands to their subordinates upon a reboot.

We've relieved the stall by asking LSC_CONFIGs directly to go to DOWN (even though LSC_CONFIGS itself is managed by ISC_LOCK). LSC_CONFIGS then regained control of SUS_PRM and ITMY, and they traversed their graphs to ALIGNED as expected.

We'll work with the ISC team to sort out the management confusion and flush out the proper initialization sequence.

There was some pressure excursions that may impact platform position/performance from 1608 to 1638utc on 10 March 2015.

Fluctuations of +- a few PSI were experienced--See Attached 30 minute trends of Pressure and Motor control drive.  Usually, with the servo in Auto control, a pump station can be ramped down over 300 seconds and the servo would keep the pressure up by speeding up all the other Pump Stations.  However, with the 10mhz controller parameters, it was not keeping it at set point and the controller was switched to manual and dealt with by hand.  We usually always have a glitch right at the end of the ramp down or ramp up in Pump Station output as we close/open the valve (keeping the pump from rotating backwards.)  We will examine why the pump stations could not keep these at Set point--Since this ramp down is just a abruptly starting linear ramp in motor speed, the frequency (at the start at least) was likely too fast for the servo.  Not too much harm anyway, no HEPI Platforms tripped at least.

Maintenance Performed: Pump Stations 5 & 6 (Check medm or Pump Station label) had motor grease added (2 tsp=5 gun pumps) in each of two bearings.  Accumulator pressures checked, #1 on PS was very low.  All others near nominal (60-90% of operating fluid pressure, set with no fluid pressure.)

Stopped running this morning with the following error:

Mar 10 09:05:08 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Out of range value for column 'value' at row 1: Error code: 1264: SQLState: 22003: Exiting.

All guardian nodes have been upgraded to the latest version of guardian and cdsutils:

• guardian: r1390
• cdsutils: r443

The guardian machine (h1guardian0) was rebooted and all nodes came up fine with the latest versions.

The only issue has been a problem with the latest nds2-client install that is not properly exporting the nds2-client python bindings.  So far this is only affecting OMC_LOCK.  This should be fixed momentarily.

Release notes

These changes include a couple of bug fixes and one new feature:

• work around PCASPY bug with empty char arrays, that was affecting the SPM_DIFF and USERMSG channels
• add ability to initialze node SPM setpoints
  • If a "ca_channel" attribute is provided in the system module as a list of (channel, value) tuples, it will be used to initialize the internal guardian setpoint table.

We left the IFO on a DC lock last night. Attached is the spectrum. Two nights were notable:

1) The hump around 800Hz (apparently known as "twin peaks") was bigger than ever. Interestingly it was NOT there at all during earlier locks during the day. The only thing we know we did between the locks was a complete initial alignment. Also, over the period of about 1h, the hump completely disappeared.

2) The low-frequency noise (20Hz to 100Hz) is significantly less stationary now. (This did not improve over time.)

WP #5094

The NDS2 client software has been updated to nds2-client-0.11.4 for Ubuntu 12.04 workstations.

The corner whitening filters were not switching, with an error message invalid data chn and readback different for Refl 9 and refl air 9, and for POP we had an error message invalid data for channels 1, 2, 3 and 4. 

Daneil fixed this by going to the system manager for the corner, finding the X gateway under device 1, corner MSR L0, Corner MSR L8_9 Vertex, going to the Online tab, and clicking on several of the buttons under state machine (Init, pre-op, safe op, op) in an order that seemed random to me. 

I am just writing down the solution to this problem so that we can look it up next time it happens.