model restarts logged for Tue 09/Sep/2014
2014_09_09 09:11 h1iopsusquadtst
2014_09_09 09:11 h1susquadtst
2014_09_09 09:16 h1iopsusquadtst
2014_09_09 09:54 h1iopsusquadtst
2014_09_09 09:54 h1susquadtst
2014_09_09 10:34 h1iopoaf0
2014_09_09 10:34 h1oaf
2014_09_09 10:34 h1odcmaster
2014_09_09 10:34 h1pemcs
2014_09_09 10:34 h1tcscs
2014_09_09 10:46 h1dc0
2014_09_09 10:48 h1broadcast0
2014_09_09 10:48 h1fw0
2014_09_09 10:48 h1fw1
2014_09_09 10:48 h1nds0
2014_09_09 10:48 h1nds1
2014_09_09 10:54 h1isiham4
2014_09_09 10:54 h1isiham5
2014_09_09 11:05 h1lsc
2014_09_09 11:06 h1lsc
2014_09_09 11:07 h1dc0
2014_09_09 11:12 h1broadcast0
2014_09_09 11:12 h1dc0
2014_09_09 11:12 h1fw0
2014_09_09 11:12 h1fw1
2014_09_09 11:12 h1nds0
2014_09_09 11:12 h1nds1
2014_09_09 11:21 h1isiham4
2014_09_09 11:27 h1broadcast0
2014_09_09 11:27 h1dc0
2014_09_09 11:27 h1fw0
2014_09_09 11:27 h1fw1
2014_09_09 11:27 h1nds0
2014_09_09 11:27 h1nds1
2014_09_09 14:25 h1omc
2014_09_09 15:32 h1nds1
2014_09_09 15:50 h1nds1
2014_09_09 17:04 h1iopoaf0
2014_09_09 17:04 h1oaf
2014_09_09 17:04 h1odcmaster
2014_09_09 17:04 h1pemcs
2014_09_09 17:04 h1tcscs
2014_09_09 17:22 h1iopoaf0
2014_09_09 17:22 h1oaf
2014_09_09 17:22 h1odcmaster
2014_09_09 17:22 h1pemcs
2014_09_09 17:22 h1tcscs
2014_09_09 17:34 h1iopoaf0
2014_09_09 17:34 h1oaf
2014_09_09 17:34 h1odcmaster
2014_09_09 17:34 h1pemcs
2014_09_09 17:34 h1tcscs

maintenance day. Startup of h1susquadtst. Swap of DAC in h1oaf0, subsequent replacement of original DAC later. New ISI Ham4,5 code. Reversion of h1lsc to RCG2.8.3. New code for h1omc. Associated DAQ restarts. Unexpected failure of h1nds1.

(Jeff, Jamie, Sheila, Alexa, Lisa)

Tonight we used ITMX ISI with the guardian paused. We had damping on stage 2, and on stage 1 X, RX and RY. For stage 1 we used the level 3 controllers that are loaded and a gain of 2.

Our goal for the night was to lock PRMI on the carrier. However, we would lose lock when we aligned PRM after locking MICH on mid fringe. We must be missing some setting. We had found the FM2, FM3, FM4 were turned off (probably for locking on PRMI sideband) on LSC_POPAIR_A_LF. So maybe we are missing something else, but we just don't know what. Also the counts on LSC_POPAIR_A_LF_OUT have decreased to 1.8 from 3. To handle this, we adjusted the normalization matrix and triggering values, but this did not seem to work. The MICH mid-fringe OLTF was consistent with the previous measurements. We also slowed down the ramp for aligning PRM, but this did not help.

Jamie, Lisa

Jamie improved the tool he adapted from SEI to be able to plot multiple channels and rescale the axis all at the same time. It is now ready for public consumption.
The script which does the magic lives here:/opt/rtcds/userapps/release/sys/common/scripts/lockloss.

usage: lockloss plot [-h] [-w WINDOW] [-c FILE] [-o FILE] time

plot specific time

positional arguments:
  time                  GPS time to plot

optional arguments:
  -h, --help            show this help message and exit
  -w WINDOW, --window WINDOW
                        plot time window in seconds (default: '[-240, 5]')
  -c FILE, --chanfile FILE
                        file with list of channels to plot, 1 per line ('-'
                        for stdin)
  -o FILE, --outfile FILE
                        save plot to file

This is an example which uses a list of channels stored in my PRMI config file with some useful signals. I would be happy to attach this file to this entry, but apparently I can't.
In any case, it is just a list of channels...

lisa.barsotti@opsws3:~$ lockloss -c ~/lockloss/PRMI plot -w '[-200, 5]' 'tconvert Sep 9 10:34 UTC'

This is the lock loss that Kiwamu reported here , where the sideband build up is dominated by a 40 mHz oscillation in ITMX YAW.
The power fluctuations are kind of huge (75%), so the fact that the PRMI unlocked is not surprising, as Kiwamu was saying.

This tools will then talk to the Guardian and automatically provide a selectable list of lock loss GPSs.

The message of this entry is that, as long as NDS2 cooperates, this tools drastically improves the way we have been doing lock loss analysis in the last decade (and any analysis which requires zooming multiple channels, really).

AS Jason, Keita and I disscussed this morning, I left the BS oplev damping off for several hours today, from 9/9/14 17:00 UTC to 9/9/14 23:00

Sheila, Alexa, Lisa, Kiwamu

Today we went back to locking SRY as described in 13726, to try make sure we have some reasonable crossovers for the SRC actuation.

First we adjusted the phase for AS_45, to -71.9.  We also had to flip the sign, so we are now locking with a gain of -400 in SRCL. 

we are using a gain of 1 in SRM and SR2 ISC INF. 

After talking to Ryan we moved the 27 Hz notch to the SRM M2 lock filter, and disengaged the 27Hz notch in SRM ISCINF.  This keeps the DAC from saturating at the 27 Hz vertical mode, without causing a notch in the open loop gain.  

We have a ugf of about 30 Hz, measurements in the SRM path and the SRM M2 path are also attached.  It seems that we do not ring up the vertical mode in SR2 as easily as we ring it up in SRM. 

Causes outer "ability to drive" boarder to go white on overview screen.
H1:ISI-ITMY_ST1_WD_MON_STATE_INMON 
H1:ISI-ITMY_ST1_MASTER_SWITCHMON 
H1:ISI-ITMY_DACKILL_STATE         <-- This one shows red on middle click.
H1:IOP-SEI_B1_DACKILL_STATE

[This work was done last week, but I forgot to submit this log.]

I added NOMINAL state definitions for all SEI systems, including HPI, ISI stages, and HAM and BSC chamber managers.  All SEI guardian nodes were restarted to affect these changes.

NOMINAL state for HAM managers is "ISOLATED".

NOMINAL state for BSC managers is "FULLY_ISOLATED" (HPI and both ISIs isolated).

NOMINAL state for HPI is "ROBUST_ISOLATED", for both HAMs and BSCs.

NOMINAL state for ISI stages is "HIGH_ISOLATED".

Other than HAM 5 and 6, which are still being commissioned, everything except BS is set to be in the NOMINAL state, and is in the NOMINAL state.

We are currently operating the BS ISI with only stage 1 isolated, and stage2 left damped, corresponding to "ISOLATED_DAMPED" in the BS chamber manager (MICH locking has a tendency to trip the ISI stage 2 if isolated).  However, I decided to leave the NOMINAL settings to the default for ISI_BS_ST2 and SEI_BS, so that we're reminded to address this issue down the line.  Here's a crop from the GUARD_OVERVIEW, showing that the SEI_BS and ISI_BS_ST2 are not in their nominal states, as indicated by the far left OK indicators being orange:

We still need to add numeric indices for all the SEI states.  This will require a bit more work for SEI since the HPI and ISI states are generated based on the particular configuration.

J. Kissel, J. Rollins, S. Dwyer, A. Staley

While Sheila and Alexa began to lock PRMI on carrier, the HAM2 and HAM3 HEPI and ISIs tripped for an unknown reason. We'll leave this to people offline to figure out what happened. See attached actuator trip plots from the ISIs.

model restarts logged for Mon 08/Sep/2014
2014_09_08 12:41 h1fw0

unexpected restart of h1fw0.

As the title says.  I replaced the ITMy oplev laser today and there is no change.  Data looks exactly the same as in Keita's previous alog.  Will continue to investigate tomorrow.

GV20 to have been closed next week anyway for X2 pressure accumulation 

Found h1nds1 had died about 15:49 PDT.  Rebooted computer, as it was unresponsive.  Note that the daqd process for h1nds1 had restarted itself at 15:30 PDT.

08:04 Karen - mopping at exit door in LVEA.

08:21 Aaron and Fil to LVEA to pull power cables for Illuminators and cameras from HAM3 F rack to Biergarden. Also, install junction box on top of HAM3 field rack. Drilling will be done and vacuums will be used.

08:25 Hugh to End stations to check on HEPI pumps.

08:47 Fire Dept on site to do annuals on fire extinguishers

09:02 D Barker - out to LVEA to start the Quad tst stand FE

09:59 Quad tst stand back up. Dave and Jim to restart OAF

10:11 Cris to End-X 

10:20 Richard out to LVEA by Fil

10:35 OAF is back up

10:47 J Kissell taking down HAM4,5 ISI to recompile model to include OpLev

10:57 Richard out of LVEA

10:58 P King into H1 PSL enclosure to execute work permit 

11:02 Hugh back from end stations

11:06 Jason out o LVEA to swap out ITMY oplev LASER

11:07 D Barker - DAQ restart

11:11 Praxxair called 10 minutes ahead of delivery - en route

11:55 Jason out for AHM 

12:00 All HANDS MEETING

13:43 Jason back into LVEA -replace bad power supply at IMTY oplev

14:27 Alistair into LVEA to "play with LASERS" - his words.

14:50 Mitch into the LVEA

15:00 Mitch out of the LVEA

15:45 Jason out of LVEA - apparently successful

BrianL Hugo JimW Hugh

When I collected HAM4 TFs 3 Sept, a deep zero here seen in the first plot as compared to the second, caught our eyes and it looks to be a problem possibly in the HEPI Isolation loop.

Here is the transverse controller from HAM3 4 & 5 in the attached.  The HAM3 controller is the top graph, appears to be done by hand and at a UGF of 2hz.  The middle graph is HAM4 and are the HAM HEPI Generic Loops developed by Hugo w/a ugf of 5hz and implemented by Hugh.  The bottom graph is HAM5 and again like HAM4, are generic and installed by me.  It is surprising that the HAM4 looks more of an issue than the very sharped peaked HAM5; but, the shoulders of the peak are higher at the ISI zero for HAM4 and the zero at HAM5 is below 6hz rather than above.  That said, the HAM5 HEPI controller is obviously equally problematic given this gain peaking.

Bottom lines--Generic loops while easy to install, should be carefully checked

Is a 5 hz loop best, warrented, needed?

Post HEPI turn-on ISI TFs should be collected and thoroughly checked.

Apologies to my above co authors--they have not been consulted for this entry.

Yesterday people found that PR3 and BS seemed to move when PRC is carrier locked.

For PR3, there's no feedback and we see a clear correlation between the carrier power (sorry the first plot doesn't have the carrier power, only the AS power) and both OL and witness OSEMs. All in all PR3 oplev seems to slowly go negative in PIT after the carrier lock and goes back positive after lock loss, though this doesn't completely describe the behavior shown on the first plot.  PR3 DC OPLEV servo should be able to counter this if needed.

For BS it's somewhat more difficult to see partly because of the length feedback, but there's definitely a correlation between the carrier buildup and the BS oplev.  Unlike PR3, every time there's a carrier lock OPLEV  PIT goes positive and slowly comes back. BS oplev doesn't seem to be well correlated to the mean length feedback, so this is not the effect of the DC length push tilting the BS.

WP4834 Jim and Dave

We replaced the 16bit DAC card in h1oaf0 (PCIX card on PCIe adapter) with a PCIe DAC card. This is the second DAC card in h1oaf0, the first is an 18bit DAC. Procedure was:

• stop running models
• take unit out of Dolphin fabric (local command)
• power down CPU
• power down IO Chassis
• replace card
• power up IO Chassis, power up CPU

This does not close out the WP, we have SUS EY and SEI EX still to complete.

Trying to get this in before the SEI T&C call. Pictures are from open loop measurements of ITMX. Foton pics are comparing T240 blends and X loops from ITMY and ITMY.