In loading the new code for the SR3 cage servo, I caused the guardian to go into error (because there is a self.counter that is initalized in main and used in run). This took us out of observing.  I took the CAGE_SERVO guardian to manual, turned it off and back on so that it would initalize the counter.  Then I attempted to INIT ISC_LOCK so that it would manage the CAGE_SERVO again. The cage servo is managed by the DRMI guardian, though.  I took DRMI to manual, init, this broke the lock for some reason that I don't understand at all.

Apologies.

TITLE:  12/16 EVE Shift:  00:00-08:00UTC (16:00-00:00PDT), all times posted in UTC     

STATE of H1:   NLN just under 80Mpc

Outgoing Operator:  Jim

Quick Summary:

Walked in to find film crew in the middle of an interview, so held off entering the Control Room for a while, but Jim had H1 with still running from the lock I started from last night.  Once I was able to squeeze into the Control Room, after a while, H1 had a lockloss, and commissioners noted that that this was probably due to an "EPICS Freeze".  As far as I know, an "EPICS Freeze" symptom is that all the StripTool traces flatline & thus we appear to lose the ability to control H1 and thus have a lockloss.  For this one, the flatline happened on the order of 80sec and then we had the lockloss.  NOTE:  yesterday people mentioned "EPICS Freezes" last night, but they were shorter and we stayed locked.

Filming continues!  :)

Dave, Sheila

At 00:47 UTC, we lost lock after a 45 second epics freeze.  The attached screenshot shows that the cage servo stopped updating, durring the freeze, which is the intended behavior since we modified the guardian (22754) to watch and not update the servo when the PITMON does not update. 

Dave looked up the guardian logs, and indeed the guardian does have the intended behavoir.  However, when the servo comes back on after being off this long it puts out a large number.  Now we have added a counter that gets incremented when the PITMON has not updated, and cleared when it is updated.  If PITMON has not updated in 16 cycles, the guardian will go to CAGE_SERVO_OFF, then should return to CAGE_SERVO_RUNNING and reinitalize the servo.  This will happen once a second until the epics freeze is over. 

O1 days 88-89 

model restarts logged for Tue 15/Dec/2015
2015_12_15 08:26 h1isietmx
2015_12_15 08:28 h1isietmy
2015_12_15 08:33 h1isiitmx
2015_12_15 08:38 h1isiitmy
2015_12_15 08:42 h1isibs
2015_12_15 08:49 h1isiham6
2015_12_15 08:50 h1isiham5
2015_12_15 08:52 h1isiham4
2015_12_15 08:55 h1isiham3
2015_12_15 09:02 h1isiham2

Maintenance day, new ISI code for both BSC and HAMS. No DAQ restart required.

model restarts logged for Mon 14/Dec/2015 No restarts reported

WP5658. Jim, Jonathan, Carlos, Dave:

The ops account home directory has been moved from /ligo/home/ops to /opshome/ops. The new location is a partition on a new 2TB disk drive on the NFS server cdsfs0. I have found three uses of this home directory:

the ops account on the operator workstation*
the VerbalAlert system
the Lock Logging system

Operator0's local ops account was changed to use the new home directory. Corey is testing this by logging in as ops. The verbal alert system was changed earlier, see alog. The lock logger runs on nuc0, this code was modified to use the new path and the source code is now ran out of the new location.

To prevent any unknown systems from writing to the old location, I have made /ligo/home/ops read-only.

* the operator workstation was a "loaner machine" called opsws9. Today Carlos and Jonathan permanently renamed it operator0.

we had a large epics freeze-up event which knocked H1 out of lock. The duration of the freeze was 16:45:46 - 16:46:56 PST.

TITLE:  12/16 Day Shift:  16:00-0:00UTC

STATE of H1: Observing ~80 Mpc

Support:  Normal Control Room crowd

Quick Summary: Quiet day, welcome break after yesterday

Activities:

16:00 JeffB to Mech Room

18:30 RichM to Mech Room

19:00 John, Kyle, MikeZ to Y mid

23:30 Chris Biwer setting up and running hardware injections

I'm preparing to do a set of hardware injection tests. I will update this aLog entry as injections are scheduled.

I first need to svn up the repo to get the injection files. And check that the latest version of tinj is installed.

**Short version: Increased RY input motion (maybe HEPI, maybe wind/ground) causes ISI X loops to ring up when running 45mhz blends. The suspension/tidal is not the cause. The 90mhz blends seem to be immune to this. Other than using 90mhz blends, I'm not sure how to fix the ISI's configuration, short term to prevent the ISI from ringing up. But we should put a StripTool of the end station ISI St1 CPS locationmons somewhere in the control room so operators can see when ground tilt has rung up an ISI. Alternatively, we could add a notification to VerbalAlarms or the DIAG node when an ISI has been moving something like 10 microns peak to peak for several minutes.

This morning while the IFO was down for maintenance, Evan and I looked at ETMX to see if we could figure out what is causing the ISI to ring up. First we tried driving the L1 stage of the quad to see if some tidal or suspension drive was the cause. This did not have on the ISI, so I tried driving on HEPI. When I drove HEPI X, the ISI rang up a bit, but no more than expected with the gain peaking of the 45mhz blends. When I drove HEPI in RY, however, the ISI immediately rang up in X, and continued to ring for several minutes after I turned the excitation off. The attached image shows the ISI CPS X(red), RY (blue), HEPI IPS RY(green) and X (magenta). The excitation is visible in the left middle of the green trace, also visible in the sudden increase in the red trace. I only ran the excitation for 300 seconds (from about 1134243600 to 1134243900), but the ISI rang for twice that. After the ISI settled down I switched the blends to the 90mhz blends and drove HEPI RY again. The ISI moved more in X but it never rang up even after I increaed the drive by a factor of 5. The second plot shows the whole time series, same color key. The large CPS X motion (with barely noticeable increase in the IPS RY) is the oscillation with the 45mhz blend , the larger signal on the IPS RY (with small increase in CPS X) is with 90mhz blends. The filter I used for each excitation was zpk([0 0],[ .01 .01 .05 .05], 15111).

the Verbal Alarms code was logging to the ops home directory. Prior to the move of this home directory (WP5658) I have modified the code to log to a new directory:

/ligo/logs/VerbalAlarms

We restarted the program at 14:04 and verified the log files are logging correctly.

Attached are two side by side 10 hour trends during observation mode w/ Quite_90 blends on Z in the left group and 45mHz Z blends in the right group.  These are about four days apart, 8 vs 12 Dec.  I've scaled each plot to be the same on the two trends.

The first four signals are X & Y Ground motion and the seismic environment is a mix with some better others worse in the second group.  The ASC signals are very close with only suttle differences.  We need to look at spectra now but this says the ASC doesn't care or this yardstick is too coarse.

Not sure if anyone has already caught this. Switching the ETMX blend to Quiet_90 on Dec 14th caused glitches to appear around 20Hz region (Fig 1 - starting at 20:00:00 UTC.) while switching both ETMX and ETMY to Quiet_90 everywhere caused gliches to appear around 10 Hz and 30 Hz region (Fig 2 - starting at 9:00:00 UTC). Wind speed has been low (<5mph) and the useism (0.03-0.1Hz) has been around 0.4 um/s. BNS range has been glitchy since the blend was switched but the lock has been relatively more stable. The question is, do we want clean data but constantly risk losing lock when the tidal rings up, or slightly glitchy data but relatively more stable interferometer?

Following aLog 24208, all LHO ISI platforms were restarted.  Most platforms deisolated nicely.  But, HAM2 decidedly did not.  New safe.snaps were captured for all platforms.  This captured the FF paths being off at start up and GS13 being in low gain.  Guardian was adjusted for HAM2 to desable the GS13 gain switching.

All snaps and isi/h1 guardians were committed to the svn.