Corey suggested looking at DIAG_SDF log, and there is activity that coincides with H1 going out of Observe:

19:19:45UTC - H1 out of Observe, and DIAG_SDF shows:

• 2016-12-10T19:19:44.81946 DIAG_SDF [RUN_TESTS.run] USERMSG 0: DIFFS: sysecaty1plc2: 1
• 2016-12-10T19:19:47.78952 DIAG_SDF [RUN_TESTS.run] USERMSG 0: DIFFS: sysecaty1plc2: 1
• 2016-12-10T19:19:52.26944 DIAG_SDF [RUN_TESTS.run] USERMSG 0: DIFFS: sysecaty1plc2: 1
   

Now, how do I know what "USERMSG 0: DIFFS: sysecaty1plc2: 1" is?

Keita's alog 32134 - instructions on how to look for channels that changed

My bad - investigating - looked at SDF - kicked H1 out of Observe:

• 20:08:47UTC - H1 out of Observe
• 20:09:21UTC - H1 back in Observe
• no change to H1 config

DIAG_SDF log:

• 2016-12-10T20:08:44.09793 DIAG_SDF [RUN_TESTS.run] USERMSG 0: DIFFS: sysecatc1plc2: 4

• I ran the scripts in Keita's alog 32134 and did not find the channel that kicked H1 out of Observe
• emailed Keita
• He wrote some new files to hunt for the channel that kicked H1 out of Observe
• those files are in /his directory in LockLoss/SDFERROR
• command is
  • > for ii in SDFLIST*.txt; do lockloss -c ${ii} plot -w '[-10, 10]' gpstime; done
  • or this event I used gpstime = 1165432802
• channel that kicked H1 out of Observe is
  • H1:FEC-1031_SDF_DIFF_CNT
  • it toggled 3 times and that agrees with what I found in the DIAG_SDF log
• the next step is to identify to Front End, but middle mousing on the SDF diff count
  • the Front End responsible is EY ECAT PLC2
  • is it possible that sysecaty1plc2 is sys-ecat-y1-plc2?

• Email from Keita about searching for the exact channel(s) that took H1 out of Observe.
• I ran them once and didn't see a clear plot of which channel.
• Corey's going to run them again and see if he comes up with something different.

From Keita:

I took
/opt/rtcds/lho/h1/target/h1sysecaty1plc2sdf/h1sysecaty1plc2sdfepics/OBSERVE.snap

and stripped unnecessary information, split into 20 line chunks and
put them here:
/ligo/home/keita.kawabe/LockLoss/SDFERRORS/h1sysecaty1plc2

Could you again run the lockloss tool by
for ii in ecaty1plc2*; do lockloss -c ${ii} plot -w '[-10,10]' gpstime; done
 

This morning (Monday Dec 12) I ran the lockloss script and I can see that H1:ALS-Y_FIBR_LOCK_TEMPERATURECOMTROLS_ON was flipping (see attached, second column from the left). Other things like LASER_HEAD_CRYSTALFREQUENCY, CRYSTALTEMPERATURE and VCO_TUNEOFS were also changing but these were not monitored.

Anyway, it's strange that this was not found when Cheryl and Corey ran lockloss tool. Maybe NDS2 was misbehaving?

Just to make sure, what I did is:

cd /ligo/home/keita.kawabe/LockLoss/SDFERRORS/h1sysecaty1plc2

for ii in ecaty1plc2_a*; do lockloss -c ${ii} plot -w '[-10, 10]' 1165432802; done