TITLE: 08/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 51Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    Wind: 4mph Gusts, 2mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY: locked 24+ hours

TITLE: 08/14 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1:
INCOMING OPERATOR: Travis
SHIFT SUMMARY:

"Seismon system not updating" DIAG_MAIN alarm, we aren't getting updates of 4.0+ EQs (last one was at 3:05utc, but we've had others since).
H1 has been locked for over 24hrs with a steady 52Mpc.
LOG:

• 8:22 - 8:33  Vehicle at LSB, person walked toward LSB entrance & then drove off.

Laser Status:
SysStat is good
Front End Power is 33.88W (should be around 30 W)
HPO Output Power is 154.7W
Front End Watch is GREEN
HPO Watch is GREEN

PMC:
It has been locked 5 days, 18 hr 52 minutes (should be days/weeks)
Reflected power = 17.86Watts
Transmitted power = 57.96Watts
PowerSum = 75.82Watts.

FSS:
It has been locked for 1 days 0 hr and 15 min (should be days/weeks)
TPD[V] = 1.072V (min 0.9V)

ISS:
The diffracted power is around 2.3% (should be 3-5%)
Last saturation event was 1 days 0 hours and 54 minutes ago (should be days/weeks)

Possible Issues:
PMC reflected power is high

TITLE: 08/14 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 51Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    Wind: 10mph Gusts, 8mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s

Small step up with useism (factor of 2 over last 24hrs).  Winds up & down over last 24hrs (currently quiet).
QUICK SUMMARY:

While going through checksheet, noticed a dead video4 computer (rebooting).  Other than that, no notable changes.

TITLE: 08/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 52Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Not much happened, locked entire shift
LOG:
Around 4:00 UTC, I saw that our seismon code was dead. Not sure what happened, but there was a prolonged period without any 4.0+ earthquakes. Maybe some of the cleanup processes broke it (i.e. all event folders were wiped, seismon couldn't decide where to new ones) ?

TITLE: 08/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 52Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY:  Locked the entire shift, 8.5 hours total now.  No issues to report.
LOG:  Restarted Video5.

Lock is 5 hours old.  We rode through one/two EQs that, interestingly enough, were ONLY reported by SEISMON.  See screenshots.

TITLE: 08/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 50Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 11mph Gusts, 9mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:  No issues handed off.  Lock is 30 minutes old.

TITLE: 08/13 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 51Mpc
INCOMING OPERATOR: Travis
SHIFT SUMMARY:

H1 locked most of the shift. "Details" of lockloss in previous aLog. Fire reported in transition log seems to be out. There was some rain. Handing off locked/Observing H1 to Travis.
LOG:

12:58 Lockloss. Doesn't seem to be environmental and so far can't see anything else responsible.

13:39 Initial Alignment

14:38 H1 back to Observing.

A few failed attempts at re-locking prompted the necessity to do an initial alignment. Initial alignment went well. The sequence was stopped at DC readout to damp violin modes ETMX 2&4.

TITLE: 08/13 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 50Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    Wind: 22mph Gusts, 18mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

I was briefed on what would happen when if +/-18VDC supply at EX trips. Watching the fire burning. It looks to me as if it's way off in the distance to the NE across the river. Gard to tell out here in the dark but the alight scent of smoke is in the air outside. The wind is out of the SW.

TITLE: 08/13 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 50Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY: Problems with sus power supplies and noisy earth. And fires.
LOG:

When I arrived Patrick was still trying to diagnose a tripped ETMX sus. Jeff has a alog on that. 

4:00 RIchard on site to replace ETMX power supplies. I  put in alog about that. Around the same time an Indonesian earthquake arrives.

6:58 Finally back to observing.

Richard said he thought he saw a fire near the Y barricade when he left. I couldn't find it on the cam, so I went to the roof and sure enough, looks like there's a fire along Highway 10, north of the site. Attached is a screen cap from the roof cam. Not much to see, the fire is still a ways out. Couldn't see if HFD was doing anything about it.

Looks like the ETMX power supply has tripped again. I called Richard, he came out and has replaced the power supply. He replaced +18 power supply s1202020 with s1300282.

While he was coming in I looked at the temps in the rack areas at EX, they've been steady for the last 3 days at least.

Unfortunately, right after I got off the phone with Richard, I got notification for a 6.5 eq in Indonesia. Looks like I won't be locking for an hour or so.

J. Kissel, J. Warner, P. Thomas (R. McCarthy and D. Barker remotely)

I happened to be on site giving this Saturday's tour, for better or worse, when Jim and Patrick informed me that End X had some sort of electronics failure that took down everything (see first entry here: LHO aLOG 382517). 

After a debrief from both of them while driving to the end station, we knew to start our investigation at the SUS rack power supplies: the symptom, they'd informed me after having already gone to the end station, that the coil drivers and AA/AI chassis in the SUS rack showed only one leg one via the LEDs in the back. 
We don't know why this happened, and have not investigated.
As a consequence, however, this explains why the OSEM sensor report to the independent software watchdog went dead, and thus tripped that watchdog, killing the seismic system via watchdogs.

Upon arrival, we went immediately to the power supplies, and indeed found the left leg of the 18V power supply with its ON/OFF rocker switch in the OFF position. We turned it back ON, it came to life  (see "after" picture IMG_2234.jpg), and this restored the SUS electronics and its OSEMs (see "after" picture IMG_2235.jpg).

However, unfortunately, we also noticed that the EX:ISC 12V power supply LED lights were flickering (see video on YouTube because .m4v format is not accepted by the aLOG). Not knowing that this was a normal thing (we found out later once we got a hold of Richard), we also power cycled both legs of those ISC 12V power supplies hoping to clear the blinking. 
Sadly
(a) this did not clear the blinking,
(b) it killed the power to the neighboring IRIGB timing system, which informs (and thus killed) the end station ISC I/O chassis. That, in turn, killed the entire end station Dolphin Network.
(c) (we found out later) it killed the power supply to the relay that feeds the power to the ESD HV. Namely, once we got up to looking at the ESD HV power supply (we worked our way up the rack from the bottom) we found that with rocker switches ON, there were no displays and OFF, and they were unresponsive to flipping the ON/OFF rocker switches (as pictured in IMG_2233.jpg).

In order to fix (b) we got a hold of Dave, who is having to restart all front-ends and IO chassis remotely. He'll likely aLOG later.

In order to fix (c) we followed power cables to the HV power supplies up to the top of the power supply rack, where found what we now know as a relay in the state as pictured in IMG_2236.jpg and IMG_2237.jpg.
Eventually, we found the big red button on the "front" of the chassis (see IMG_2238.jpg), and like all good red buttons, it drove us to push it (cautiously, we had left the ESD power supplies in the OFF position before doing so, however). We heard a few clicks, and all the green lights lit up. Once that was happy, then the ESD HV power supplies came back to life with a simple flick of the power switch.

As of now, Jim is re-isolating tables, and resuming recovery with a fully functional system. "Normal" recovery from here, but we should be cautious of violin modes.

Again, we have not invested why the power supply tripped off in the first place; we'll leave that for the work week or for those more curious than us.

following the reset of the EX 12V DC supply which power cycled the timing fan-out, I have restarted all the models at EX using the sequence /etc/kill_models.sh followed by /etc/start_models.sh

I monitored the IRIG-B status on each IOP, they all look good. After the models were running again, I cleared the IPC errors and the CRC counters.

TITLE: 08/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Unknown
INCOMING OPERATOR: Jim
SHIFT SUMMARY: No issues until lock loss at 21:49. Currently working with Jim to diagnosis ETMX.
LOG:

17:00 UTC Headed down to LSB to do a fire check, ran into Amber driving up. She said it was fine, so I went back to the control room.
19:00 - 19:18 UTC LSB fire check
~21:00 - 21:09 UTC Greg leading tour through control room, then to overpass
21:13 - 21:27 UTC TJ leading tour through control room, then to overpass
21:49 UTC Lock loss (see previous alog)

Lost lock at 21:49 UTC.

Verbal alarms reports 'ETMX RMS WD has tripped'.

DIAG_MAIN reports:
SUS_RMS_WD: ETMX RMS ['UL', 'LL', 'UR', 'LR'] WD has tripped
SUS_WD: SUS ETMX M0 osem filter input inmon is dead
OPLEV_SUMS: ETMX sum is low

It seems that the hardware watchdog may also have tripped, at least according to the MEDM screen. H1:SUS-ETMX_HWWD_STAT_OUT has jumped to 15.

I have not been able to reset the PUM COIL SLOW CUR WD using the WD RESET button.

The wiki says a trip to the end station may be needed. I have not yet found the procedure for what to do at the end station.

Screenshots attached.

   Could not reach Pcal folks to confirm SDF differences. Accepted the differences (see file below) to get back to Observing.