Ryan has  tracked down the core problem. With older kernels, the sched clock overflows in about 208.5 days. My previous alogs showed that the computers have been running since the last power outage (30 Sep 2016 06:45), and the error messages on h1susex and h1seiex suggested a clock zeroing late Wednesday night (26 Apr 2017 22:00). The time difference between these two times is 208.6 days.

We are investigating why only two computers have crashed, why both at EX, why both around 6am and why one day apart.

If you google 'sched clock overflows in 208 days' you will see many articles on this. One article references kernel 2.6.32, we are running 2.6.34.

Was it reported by Terramon, USGS, SEISMON? Yes, Yes, Yes

Magnitude (according to Terramon, USGS, SEISMON): 6.8 or 7.2, depending on when you looked

Location: 30km SW of Burias Philippines

Starting time of event (ie. when BLRMS started to increase on DMT on the wall): ~20:45 UTC

Lock status? L1 & H1 lost lock at nearly the same time, LLO only a few minutes after us.

EQ reported by Terramon BEFORE it actually arrived? Yes
 

seismon reported this ~10 before it arrived and we got the Verbal notification,  we lost lock probably on the P wave arrival. After lockloss and .03-.1 hz blrms went over 1 micron, I switched the seismic configuration to the large eq configuration.

The clean room that will be used for the up coming vent has been relocated to the beer garden. A little fine tuning will be required on 5/8.
The crane is still hooked up to the clean room and the remote control is in my office.

At the time the h1seiex and h1susex computers locked up, the last errors to be printed to the console were photographed and attached to the alog. The timestamps (running seconds since last reboot) shown are not consistent with time since last boot.

Looking at the boot logs on h1boot, here are the times of the last two boots of these computers (including this week's boots, all times are local)

A computer running since 9/30/2016 should have a running clock of about 18 million seconds. The times shown in the photographs are small. Since we know at which time the computers froze, we can extrapolate back to the apparent zero time (all times local)

the "pseudo" boot time when the clock was zeroed are within two hours of each other Wednesday night.

[JeffK, JimW, Jenne]

In hopes of making some debugging a bit easier, we have updated the safe.snap files in SDF just after a lockloss from NomLowNoise. 

We knew that an earthquake was incoming (yay seismon!), so as soon as the IFO broke lock, we requested Down so that it wouldn't advance any farther.  Then, we accepted most of the differences so that everything but ISIBS, CS ECAT PLC2, EX ECAT PLC2 and EY ECAT PLC2 (which don't switch between Observe.snap and safe.snap) were green. 

Jeff is looking at making it so that the ECAT models switch between safe and observe snap files like many of the other models, so that ISIBS will be the only model that has diffs (21 of them). 

Note that if the IFO loses lock from any state other than NLN, we shouldn't expect SDF to all be green.  But, since this is the state of things when we lose lock from NLN, it should be safe to revert to these values, in hopes of helping to debug.

Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 1596 seconds. LLCV set back to 17.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 812 seconds. TC A did not register fill. LLCV set back to 41.0% open.

H1 went into observation mode at 16:58:18 UTC, the startup of the CW calibration on h1calex completed at 16:59:48 UTC

model restarts logged for Thu 27/Apr/2017
2017_04_27 07:07 h1hpietmx
2017_04_27 07:07 h1iopseiex
2017_04_27 07:07 h1isietmx
2017_04_27 07:21 h1iopsusex
2017_04_27 07:21 h1susetmx
2017_04_27 07:23 h1hpietmx
2017_04_27 07:23 h1iopseiex
2017_04_27 07:23 h1isietmx
2017_04_27 07:23 h1susetmxpi
2017_04_27 07:23 h1sustmsx
2017_04_27 07:25 h1alsex
2017_04_27 07:25 h1calex
2017_04_27 07:25 h1iopiscex
2017_04_27 07:25 h1iscex
2017_04_27 07:25 h1pemex

Unexpected crash of h1seiex, also restarts of models on h1susex and h1iscex.

model restarts logged for Mon 24/Apr/2017 - Wed 26/Apr/2017 No restarts reported

J. Kissel, K. Izumi, J. Warner, S. Dwyer

After another ETMX front-end failure this morning (see LHO aLOG 35861, 35857 etc.), the recovery of the IFO was much easier, because of yesterday morning's lessons learned about not running initial alignment scripts that suffer from bit rot (see LHO aLOG 35839). 

However, after completing recovery, the SDF system's OBSERVE.snap let us know that some of the same critical initial alignment references were changed at 14:17 UTC, namely 
- the green ITM camera reference points:
     H1:ALS-X_CAM_ITM_PIT_OFS
     H1:ALS-X_CAM_ITM_YAW_OFS
and 
- the transmission monitors red QPDs:
     H1:LSC-X_TR_A_LF_OFFSET

After discussing with Jim, he'd heard the Corey (a little surprisingly) didn't have too much trouble with turning on the green ASC system, which, if these ITM camera offsets are large, then that means the error signals are large, and we'd have the same trouble closing them as yesterday.

We traced down the change to when Dave had to the reboot of h1alsex & h1iscex this morning at around 14:15 UTC -- see LHO aLOG 35862 -- and those two models out of date safe.snap files restored. Recall that the safe.snaps for these computers are soft linked to the down.snaps in the user apps repo:
    /opt/rtcds/lho/h1/target/h1alsex/h1alsexepics/burt ]$ ls -l safe.snap 
    lrwxrwxrwx 1 controls controls 62 Mar 29  2016 safe.snap -> /opt/rtcds/userapps/release/als/h1/burtfiles/h1alsex_down.snap

    /opt/rtcds/lho/h1/target/h1iscex/h1iscexepics/burt ]$ ls -l safe.snap
    lrwxrwxrwx 1 controls controls 62 Mar 29  2016 safe.snap -> /opt/rtcds/userapps/release/isc/h1/burtfiles/h1iscex_down.snap
where the "safe.snap" in the local "target" directories are what the front uses to restore its EPICs records (which is why we've intentionally commandeered the file with a soft link to a version controlled file in the userapps repo).

We've since reverted the above offsets to their OBSERVE values, and I've accepted those OBSERVE values into the safe.snap / down.snap and committed the updated snap to the userapps repo. In the attached screenshots, the "EPICS VALUE" is the correct OBSERVE value, and the "SETPOINT" is the errant safe.snap. So, they show what I've accepted as the current correct value.

Summary: no apparent change in the induced wavefront from the point absorber.

After a request from Sheila and Kiwamu, I checked the status of the ITMX point absorber with the HWS.

If I look at the wavefront approximately 13 minutes after lock-aquisition, I see the same magnitude of optical path distortion across the wavefront (approximately 60nm change over 20mm). This is the same scale of OPD that was seen around 17-March-2017.

Note that the whole pattern has shifted slighlty because of some on-table work in which a pick-off beam-splitter was placed in front of the HWS.

TITLE: 04/28 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: LOCKING by Jim, but still in CORRECTIVE MAINTENANCE  (will write an FRS unless someone else beats me to it again!)
INCOMING OPERATOR: Jim
SHIFT SUMMARY:

Groundhog's Day shift, with H1 fine for the first 6hrs and then EX going down again (but this time SUS...see earlier alog).  This time I kept away from even breathing on the TMSx dither scripts & simply restored ETMx & TMSx to their values before all of the front end hub bub of this morning.  I was able to get ALSx aligned & this is where I'm handing off to Jim (he had to tweak on ALSy & see that he is already tweaking up a locked PRMI.  Much better outlook than yesterday at this time for sure!
LOG:

• 9:14-10:14 GRB Stand down
• 10:10 rebooted video4 (it was froze)
• 12:44 Out of OBSERVING due to EX Frontend computers going down.
• 13:43-13:55 Bubba taking pics in LVEA

H1SUSEX front end computer and IO chassis were rebooted this morning to deal with the issue posted by Corey.



Richard / Peter

(I'm almost cutting & pasting from what happened almost 24hrs ago!  Yesterday it was EX SEI, Today it is EX SUS!)

At 12:44utc (5:44amPDT):

•  Received Verbal Alarms:  
  • TypeError with ETMX RMS WD,
  • SUS ETMX watch dog tripped,
  • SUS ETMX SW watchdog tripped
  • SUS TMSX SW watchdog tripped
  • SUS TMSX watch dog tripped 
• CDS Overview WHITE for H1IOPSUSEX, H1SUSETMX, H1SUSTMSX, SUSETMXPI
• All through this, ETMx SUS & TMS systems remained damped and H1 is still in Nominal Low Noise, but am OUT OF OBSERVING.
• Flagging this as CORRECTIVE MAINTENANCE
• Called LLO Operator (Doug L) to notify them of our status.  
• Will be down on the order of hours (ie. trip to EX to restart frontends & other work)
• Will make sure NOT to touch the DITHER scripts!  (thanks, for the alog to Jeff & DAY crew)
• Could not ssh into h1susex
• Richard just arrived and we'll run through what we did yesterday morning.  I will let him contact Dave when he's ready.

(attached is a screenshot of all the WHITE screens we have for EX.)