Fairly quiet.  Two glitches on H1.

See BLRMS 0.03-0.1Hz seismic elevating.

• Terramon shows an earthquake from Colombia with an R-wave arrival earlier (at 10:36utc), which might correspond with BLRMS activity.
• H1SEISMON has an ORANGE box for the R20-Wave with an arrival of (after tconvert-ing gps time...can't we make these UTC times??) 11:56utc.  So we'll see what happens.

TITLE: 04/29 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
    Wind: 5mph
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s (at 50 percentile)
QUICK SUMMARY:

Nutsinee mentioned glitches in this current lock, but haven't had one for almost an hour.  H1 is going on 7.5hrs of being locked.  Let's see if our frontends give us grief tonight/this morning.

TITLE: 04/29 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC

STATE of H1: Observing at 66Mpc

INCOMING OPERATOR: Corey

SHIFT SUMMARY: Some oplev related tweak in the beginning of shift. No trouble relocking after the earthquake. The range seems a bit glitchy during this lock stretch. I looked at random channels but no luck so far. Detchar Hveto page doesn't have data for today yet. 

LOG:

23:40 Sheila+TJ to power cycle EX Oplev

23:55 Sheila+TJ out

00:21 Observe

00:54 Sheila going to back to EX to adjust oplev power. Switching off BRSX (forgot to go out of Observe while I did this). Out of Observe shortly after.

01:05 Sheila out. BRS turned back on. Back to Observe

Oplev glitches seem okay so far since Sheila turned the oplev power back up. Not much else to report.

We noticed on today's Hveto page that ETMX oplev is glitching.  TJ and I went out and I turned the power knob down by about 1/8th of a turn.  This reduced the output power by about 1%, and so far we don't have  glitches.  

I've used my IMC beam center measurements to calculate the change of the beam on IM1 in yaw, and propagated that change to the IO Faraday Isolator input, CalciteWedge1.  The change on IM1 is calculated using an ideal IMC (centered beams) to recent beam spot measurements from March 2017.  Nominal IM alignments are from the vent, July 2014, when the IMC REFL beam was routed through the IMs and the beam was well centered on the input and output of the IO Faraday.

My calculations show that the beam on CalciteWedge1 has moved +8.1mm, which is in the -X IFO direction, and the incident angle has changed by -1217urad, reducing the incident angle from 6.49deg to 6.42deg.

Beam Changes on IM1, IM2, and the IO Faraday input, CalciteWedge1:

The beam change on IM1 is well understood, since it comes from the IMC beam spot changes.  The IM positions can be assumed to have some error, however I've done the same calculations with IM positions from before and after the vent, and the change on CalciteWedge1 varies only by about 1mm.

A change of 8mm (+/-1mm) on the IO FI input is significant.

The optics inside the Faraday Rotator are only 20mm in diameter, and there is a small loss in aperture due to the optic mounts.

Evan G., Jeff K.

Summary:
The calibration measurement data that was collected yesterday has now been analyzed using our Markov Chain Monte-Carlo (MCMC) methods. We detail the results below. Nothing abnormal was found, and we find that the time varying factors can track changes in sensing gain and coupled cavity pole and actuation coefficients.

Details:
We analyzed the data collected during yesterday's calibration measurements, see LHO aLOG 35849. We have simplified the process for analyzing the data; rather than running two separate Matlab script to generate the MCMC results, we can now run just one script to get the final model results.

For the sensing function, we run
${CALSVN}/trunk/Runs/O2/H1/Scripts/SensingFunctionTFs/runSensingAnalysis_H1_O2.m
while for actuation, we run
${CALSVN}/trunk/Runs/O2/H1/Scripts/FullIFOActuatorTFs/analyzeActuationTFs.m.
Note that for the actuation calibration script, we have not yet converted it to an IFO agnostic script. Once it is converted, this script will be renamed.

Below are the table of values and their associated uncertainties for yesterday's measurements. Also, for comparison, are the modeled values from the reference measurement 3 Jan 2017:

                                                       Reference Value     MAP     (95% C.I.)        MAP       (95% C.I.)
                                                       2017-01-03          2017-01-03 (MCMC)         2017-04-27 (MCMC)
 Optical Gain                K_C         [ct/m]        1.088e6             1.088e6 (0.0002e6)        1.124e6   (0.0002e6)
 Couple Cav. Pole Freq.      f_c         [Hz]          360.0               360.0   (7.6)             343.4     (2.6)
 Residual Sensing Delay      tau_C       [us]          0.67                0.67    (6.7)             -1.8      (1.8)
 SRC Detuning Spring Freq.   f_s         [Hz]          6.91                6.91    (0.1)             7.4       (0.04)
 Inv. Spring Qual. Factor    1/Q_s       [ ]           0.0046              0.046   (0.016)           0.009256  (0.0069)

 UIM/L1 Actuation Strength   K_UIM       [N/ct]        8.091e-8            8.091e-8  (0.2%)          8.0818e-8 (0.18%)
 PUM/L2 Actuation Strength   K_PUM       [N/ct]        6.768e-10           6.768e-10 (0.02%)         6.795e-10 (0.08%)
 UIM/L3 Actuation Strength   K_TST       [N/ct]        4.357e-12           4.357e-12 (0.02%)         4.537e-12 (0.07%)
 UIM/L1 residual time delay              [usec]        n/a                 n/a                       29.1      (35.5)
 PUM/L2 residual time delay              [usec]        n/a                 n/a                       7.7       (3.1)
 TST/L3 residual time delay              [usec]        n/a                 n/a                       10.2      (1.8)


These values are derived from MCMC fitting to the data values. The attached plots show these results for the sensing and multiple-stage actuation functions.

We have added a new feature to the MCMC analysis, modeling the residual time delay in actuation. We expect to have zero usec of residual time delay, provided the model accurately captures all dynamics of the actuation. Deviations from zero can reveal un-modeled dynamics. For example, the PUM and TST residual time delays are inconsistent with zero usec, but we expect that this is due to imperfect modeling of the complicated violin resonances of the quad suspension.

The time varying factors are doing a good job tracking the changes between the reference model and the currently measured parameters (see time varying factors summary page).

For reference, the parameter files were used as follows:
${CALSVN}/trunk/Runs/O2/H1/params/2017-01-24/modelparams_H1_2017-01-24.conf    (rev4401, last changed 4396)
${CALSVN}/trunk/Runs/O2/H1/params/2017-04-27/measurements_2017-04-27_sensing.conf  (rev4596, last changed 4596)
${CALSVN}/trunk/Runs/O2/H1/params/2017-04-27/measurements_2017-04-27_ETMY_L1_actuator.conf  (rev4596, last changed 4596)
${CALSVN}/trunk/Runs/O2/H1/params/2017-04-27/measurements_2017-04-27_ETMY_L2_actuator.conf  (rev4596, last changed 4596)
${CALSVN}/trunk/Runs/O2/H1/params/2017-04-27/measurements_2017-04-27_ETMY_L3_actuator.conf  (rev4596, last changed 4596)

TITLE: 04/28 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 0Mpc
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:
LOG:

IFO was unlocked from EX Sus outage when I arrived, locking was a little more problematic than usual, but not bad.

18:15 Bubba to MX
18:45 Dan to MSR
22:00 Lockloss from 6.8 eq in the Phillipines
22:00 Bubba, Chandra to LVEA to close gate valves and finish craning clean room
22:45 Start locking again

Ryan has  tracked down the core problem. With older kernels, the sched clock overflows in about 208.5 days. My previous alogs showed that the computers have been running since the last power outage (30 Sep 2016 06:45), and the error messages on h1susex and h1seiex suggested a clock zeroing late Wednesday night (26 Apr 2017 22:00). The time difference between these two times is 208.6 days.

We are investigating why only two computers have crashed, why both at EX, why both around 6am and why one day apart.

If you google 'sched clock overflows in 208 days' you will see many articles on this. One article references kernel 2.6.32, we are running 2.6.34.

Was it reported by Terramon, USGS, SEISMON? Yes, Yes, Yes

Magnitude (according to Terramon, USGS, SEISMON): 6.8 or 7.2, depending on when you looked

Location: 30km SW of Burias Philippines

Starting time of event (ie. when BLRMS started to increase on DMT on the wall): ~20:45 UTC

Lock status? L1 & H1 lost lock at nearly the same time, LLO only a few minutes after us.

EQ reported by Terramon BEFORE it actually arrived? Yes
 

seismon reported this ~10 before it arrived and we got the Verbal notification,  we lost lock probably on the P wave arrival. After lockloss and .03-.1 hz blrms went over 1 micron, I switched the seismic configuration to the large eq configuration.

The clean room that will be used for the up coming vent has been relocated to the beer garden. A little fine tuning will be required on 5/8.
The crane is still hooked up to the clean room and the remote control is in my office.

At the time the h1seiex and h1susex computers locked up, the last errors to be printed to the console were photographed and attached to the alog. The timestamps (running seconds since last reboot) shown are not consistent with time since last boot.

Looking at the boot logs on h1boot, here are the times of the last two boots of these computers (including this week's boots, all times are local)

A computer running since 9/30/2016 should have a running clock of about 18 million seconds. The times shown in the photographs are small. Since we know at which time the computers froze, we can extrapolate back to the apparent zero time (all times local)

the "pseudo" boot time when the clock was zeroed are within two hours of each other Wednesday night.

[JeffK, JimW, Jenne]

In hopes of making some debugging a bit easier, we have updated the safe.snap files in SDF just after a lockloss from NomLowNoise. 

We knew that an earthquake was incoming (yay seismon!), so as soon as the IFO broke lock, we requested Down so that it wouldn't advance any farther.  Then, we accepted most of the differences so that everything but ISIBS, CS ECAT PLC2, EX ECAT PLC2 and EY ECAT PLC2 (which don't switch between Observe.snap and safe.snap) were green. 

Jeff is looking at making it so that the ECAT models switch between safe and observe snap files like many of the other models, so that ISIBS will be the only model that has diffs (21 of them). 

Note that if the IFO loses lock from any state other than NLN, we shouldn't expect SDF to all be green.  But, since this is the state of things when we lose lock from NLN, it should be safe to revert to these values, in hopes of helping to debug.

J. Kissel, K. Izumi, J. Warner, S. Dwyer

After another ETMX front-end failure this morning (see LHO aLOG 35861, 35857 etc.), the recovery of the IFO was much easier, because of yesterday morning's lessons learned about not running initial alignment scripts that suffer from bit rot (see LHO aLOG 35839). 

However, after completing recovery, the SDF system's OBSERVE.snap let us know that some of the same critical initial alignment references were changed at 14:17 UTC, namely 
- the green ITM camera reference points:
     H1:ALS-X_CAM_ITM_PIT_OFS
     H1:ALS-X_CAM_ITM_YAW_OFS
and 
- the transmission monitors red QPDs:
     H1:LSC-X_TR_A_LF_OFFSET

After discussing with Jim, he'd heard the Corey (a little surprisingly) didn't have too much trouble with turning on the green ASC system, which, if these ITM camera offsets are large, then that means the error signals are large, and we'd have the same trouble closing them as yesterday.

We traced down the change to when Dave had to the reboot of h1alsex & h1iscex this morning at around 14:15 UTC -- see LHO aLOG 35862 -- and those two models out of date safe.snap files restored. Recall that the safe.snaps for these computers are soft linked to the down.snaps in the user apps repo:
    /opt/rtcds/lho/h1/target/h1alsex/h1alsexepics/burt ]$ ls -l safe.snap 
    lrwxrwxrwx 1 controls controls 62 Mar 29  2016 safe.snap -> /opt/rtcds/userapps/release/als/h1/burtfiles/h1alsex_down.snap

    /opt/rtcds/lho/h1/target/h1iscex/h1iscexepics/burt ]$ ls -l safe.snap
    lrwxrwxrwx 1 controls controls 62 Mar 29  2016 safe.snap -> /opt/rtcds/userapps/release/isc/h1/burtfiles/h1iscex_down.snap
where the "safe.snap" in the local "target" directories are what the front uses to restore its EPICs records (which is why we've intentionally commandeered the file with a soft link to a version controlled file in the userapps repo).

We've since reverted the above offsets to their OBSERVE values, and I've accepted those OBSERVE values into the safe.snap / down.snap and committed the updated snap to the userapps repo. In the attached screenshots, the "EPICS VALUE" is the correct OBSERVE value, and the "SETPOINT" is the errant safe.snap. So, they show what I've accepted as the current correct value.

Summary: no apparent change in the induced wavefront from the point absorber.

After a request from Sheila and Kiwamu, I checked the status of the ITMX point absorber with the HWS.

If I look at the wavefront approximately 13 minutes after lock-aquisition, I see the same magnitude of optical path distortion across the wavefront (approximately 60nm change over 20mm). This is the same scale of OPD that was seen around 17-March-2017.

Note that the whole pattern has shifted slighlty because of some on-table work in which a pick-off beam-splitter was placed in front of the HWS.

TITLE: 04/28 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: LOCKING by Jim, but still in CORRECTIVE MAINTENANCE  (will write an FRS unless someone else beats me to it again!)
INCOMING OPERATOR: Jim
SHIFT SUMMARY:

Groundhog's Day shift, with H1 fine for the first 6hrs and then EX going down again (but this time SUS...see earlier alog).  This time I kept away from even breathing on the TMSx dither scripts & simply restored ETMx & TMSx to their values before all of the front end hub bub of this morning.  I was able to get ALSx aligned & this is where I'm handing off to Jim (he had to tweak on ALSy & see that he is already tweaking up a locked PRMI.  Much better outlook than yesterday at this time for sure!
LOG:

• 9:14-10:14 GRB Stand down
• 10:10 rebooted video4 (it was froze)
• 12:44 Out of OBSERVING due to EX Frontend computers going down.
• 13:43-13:55 Bubba taking pics in LVEA