ISC_LOCK, ALS_DIFF, and IFO were all showing connection errors because they lost connection to the ALS_YARM guardian, which had unceremoniously died:

2016-10-01T04:47:53.36265 ALS_YARM [INITIAL_ALIGNMENT.enter]
2016-10-01T04:47:53.36943 ALS_YARM [INITIAL_ALIGNMENT.main] ezca: H1:ALS-C_LOCK_REQUESTY => End Locked
2016-10-01T04:47:53.39004 ALS_YARM [INITIAL_ALIGNMENT.main] timer['pause'] = 10
2016-10-01T04:48:03.38190 ALS_YARM [INITIAL_ALIGNMENT.run] timer['pause'] done
2016-10-01T04:50:05.98295 ALS_YARM REQUEST: GREEN_WFS_OFFLOADED
2016-10-01T04:50:05.98317 ALS_YARM calculating path: INITIAL_ALIGNMENT->GREEN_WFS_OFFLOADED
2016-10-01T04:50:05.98360 ALS_YARM new target: OFFLOAD_GREEN_WFS
2016-10-01T04:50:06.04919 ALS_YARM EDGE: INITIAL_ALIGNMENT->OFFLOAD_GREEN_WFS
2016-10-01T04:50:06.04958 ALS_YARM calculating path: OFFLOAD_GREEN_WFS->GREEN_WFS_OFFLOADED
2016-10-01T04:50:06.04995 ALS_YARM new target: GREEN_WFS_OFFLOADED
2016-10-01T04:50:06.05074 ALS_YARM executing state: OFFLOAD_GREEN_WFS (-21)
2016-10-01T04:50:06.05096 ALS_YARM [OFFLOAD_GREEN_WFS.enter]
2016-10-01T04:50:06.05214 ALS_YARM [OFFLOAD_GREEN_WFS.main] starting smooth offload
2016-10-01T04:50:06.05215 ALS_YARM [OFFLOAD_GREEN_WFS.main] ['ITMY', 'ETMY', 'TMSY']
2016-10-01T04:50:06.55046 ALS_YARM stopping daemon...
2016-10-01T04:50:06.62930 ALS_YARM daemon stopped.
2016-10-01T04:50:07.48941 Traceback (most recent call last):
2016-10-01T04:50:07.48946   File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
2016-10-01T04:50:07.48954     "__main__", fname, loader, pkg_name)
2016-10-01T04:50:07.48959   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
2016-10-01T04:50:07.48963     exec code in run_globals
2016-10-01T04:50:07.48968   File "/ligo/apps/linux-x86_64/guardian-1.0.0/lib/python2.7/site-packages/guardian/__main__.py", line 262, in 
2016-10-01T04:50:07.49240     guard.run()
2016-10-01T04:50:07.49263   File "/ligo/apps/linux-x86_64/guardian-1.0.0/lib/python2.7/site-packages/guardian/daemon.py", line 452, in run
2016-10-01T04:50:07.49308     raise GuardDaemonError("worker exited unexpectedly, exit code: %d" % self.worker.exitcode)
2016-10-01T04:50:07.49380 guardian.daemon.GuardDaemonError: worker exited unexpectedly, exit code: -11
2016-10-01T04:50:07.61754 guardian process stopped: 255 0

As the error indicates, the worker process apparently died without explanation, which is not at all nice.

I restarted the ALS_YARM node with guardctrl and it came back up fine.  The rest of the nodes recovered their connections soon after.  As of right now all nodes appear to be funtioning normally.

This "worker exited unexpectedly" error isn't one I've seen much at all, so I'm very curious what could have caused it.

Logged an FRS (6338) for this.  Jamie was able to get everything back before ~2am, but since we canceled the OWL shift (in this pre-ER10 epoch, operators are informed to NOT wake up help in the middle of the night), instead of being down 4hrs it was more like 10hrs.  

Since this issue was fixed, the above FRS can now be closed.

For partial explanation of the guardian issue see comment to next log.

I'm guessing that the issue with the lockloss tool might have been an overlong delay finding the latest lockloss times due to excessively verbose logging of the ISC_LOCK node when it's in connection error.  This mostly exposes the weakness of the lockloss tool relying on parsing the ISC_LOCK node logs for determining lockloss times, but secondarily points to the logs being maybe unnecessarily verbose under these particular connection error conditions.  If the lockloss tool problem was *not* due to a long wait time for returning the list of lockloss times, please let me know what the error was so that I can investigate.

I have an improved version of the lockloss tool that finds locklosses much faster via NDS.  I'll push it out after I push a minor guardian update on Tuesday.  It should make the lockloss tool much faster and more robust.

I also realize there's an issue with the log display part of the lockloss tool.  This is completely orthoganal issue to the plotting, and will also be fixed with the next guardian minor release.

Very sorry about the trouble.

Input Mode Cleaner:

IMC =
    lambda: 1.0640e-06
         q: 0.2325 +13.3832i
         N: 4
      dist: [16.2406 16.2406 0.2325 0.2325 0]
      ifoc: [0.0731 0 0 0]
     label: {'MC2'  'MC3'  'Waist'  'MC1'}
       Pin: 50

z= 00.0000 m, Gouy phase: 00.0000 deg after passing MC1;   Spot size w=2.1293 mm
z= 16.2406 m, Gouy phase: 49.9133 deg after passing MC2;   Spot size w=3.3764 mm
z= 32.4812 m, Gouy phase: 99.8267 deg after passing MC3;   Spot size w=2.1293 mm
z= 32.7137 m, Gouy phase: 100.822 deg after passing Waist; Spot size w=2.129 mm
z= 32.9462 m, Gouy phase: 101.8172 deg after passing MC1; Spot size w=2.1293 mm
Round trip Gouy phase:  101.8172 deg

Gouy Phase from input coupler (MC1) to output coupler (MC3): 99.8267 deg

Input mirrors: From PSL periscope to MC3:

IM =
    lambda: 1.0640e-06
         q: -0.2325 +13.3832i
         N: 6
      dist: [0.2325 0.5507 1.0613 1.1703 1.2071 0.4130 0]
      ifoc: [0 0 0.1562 -0.3199 0 0.0409]
     label: {'Waist'  'IM1'  'IM2'  'IM3'  'IM4'  'PRMlens'}

z= 0.0000 m, Gouy phase: 0.0000 deg after passing MC3;         Spot size w=2.1293 mm
z= 0.2325 m, Gouy phase: 0.99527 deg after passing Waist;      Spot size w=2.129 mm
z= 0.78319 m, Gouy phase: 3.3516 deg after passing IM1;         Spot size w=2.1308 mm
z= 1.8445 m, Gouy phase: 7.8634 deg after passing IM2;           Spot size w=2.1444 mm
z= 3.0148 m, Gouy phase: 13.8099 deg after passing IM3;         Spot size w=1.7841 mm
z= 4.2219 m, Gouy phase: 19.9983 deg after passing IM4;         Spot size w=2.1257 mm
z= 4.6349 m, Gouy phase: 21.6763 deg after passing PRMlens; Spot size w=2.2471 mm
 

Input beam: From PSL periscope to MC3:

(The PSL periscope to input viewport distance is an educated guess.)

INP =
    lambda: 1.0640e-06
         q: -7.3931 +13.3832i
         N: 6
      dist: [0 2.7100 4.0180 0.2118 0.1400 0.2563 0.2895]
      ifoc: [0 0 0 0 0 0]
     label: {'PSLperiscope'  'INPUTViewPort'  'PeriscopeTop'  'PeriscopeBottom'  'Steer1'  'Steer2'}
 

z= 0 m, Gouy phase: 0 deg after passing PSLperiscope;                       Spot size w=2.4323 mm
z= 2.71 m, Gouy phase: 9.6308 deg after passing INPUTViewPort;         Spot size w=2.2556 mm
z= 6.728 m, Gouy phase: 26.0719 deg after passing PeriscopeTop;        Spot size w=2.1316 mm
z= 6.9398 m, Gouy phase: 26.9771 deg after passing PeriscopeBottom; Spot size w=2.1302 mm
z= 7.0798 m, Gouy phase: 27.5759 deg after passing Steer1;                Spot size w=2.1296 mm
z= 7.3361 m, Gouy phase: 28.6729 deg after passing Steer2;                Spot size w=2.129 mm
 

Power Recycling Cavity:

PRCrt =
    lambda: 1.0640e-06
         q: 7.1228 + 5.2551i
         N: 8
      dist: [16.6128 16.1551 24.8880 0 0 24.8880 16.1551 16.6128 0]
      ifoc: [-0.4391 0.0556 -2.3189e-04 -0.0010 -2.3189e-04 0.0556 -0.4391 -0.1818]
     label: {'PR2'  'PR3'  'ITMlens'  'ITMback'  'ITMlensr'  'PR3r'  'PR2r'  'PRM'}

z= 00.0000 m, Gouy phase: 00.0000 deg after passing PRM;      Spot size w=2.2471 mm
z= 16.6128 m, Gouy phase: 23.9355 deg after passing PR2;       Spot size w=6.1716 mm
z= 32.7679 m, Gouy phase: 24.8769 deg after passing PR3;       Spot size w=53.9602 mm
z= 57.6559 m, Gouy phase: 25.0458 deg after passing ITMlens;  Spot size w=52.9743 mm
z= 57.6559 m, Gouy phase: 25.0458 deg after passing ITMback; Spot size w=52.9743 mm
z= 57.6559 m, Gouy phase: 25.0458 deg after passing ITMlensr; Spot size w=52.9743 mm
z= 82.5438 m, Gouy phase: 25.2148 deg after passing PR3r;      Spot size w=53.9602 mm
z= 98.6989 m, Gouy phase: 26.1562 deg after passing PR2r;      Spot size w=6.1716 mm
z= 115.3117 m, Gouy phase: 50.0916 deg after passing PRM;    Spot size w=2.2471 mm
Round trip Gouy phase: 50.0916 deg (one-way Gouy phae: 25.0458 deg)

I see you've used 2.71 [m] as your distance between the PSL periscope an the HAM1 viewport, and mentioned this was a big uncertainty.

If I take the corner station Rack/Cable tray layout D1002704, which tells the distance between the HAM1 viewport and the +X edge of the PSL table is 86 [in], and add it to the distance from the edge of the table to the periscope mirrors from D0902114 -- 14 [in] -- I get an even 100 [in].

Thus, 2.54 [m] is likely a better number.

I tag Systems just in case they have an even more accurate/precise number, but I think past the ~inch level precision, we'd need to measure it.

T1000696-v2 has verified distances.

D1002704 is measuring PSL table North edge to HAM1 door flange, and states it's 86 inches.

D1102219-v1, Micheal Rodruck's document, aptly named H1_table_v1.pdf, has what I believe are measured values, though not in a form that's easy to use, but what I've looked at is consistant with T1000696-v2.

Using this combination of documents, the outer PSL wall to the input viewport is calculated to be 70 inches, something that could easily be measured.