Please post the device0 medm, and check the number of configured vs. actual slave modules. If the later is smaller, the following module is in error or broken.

I spoke to Filiberto on the telephone. He's heading in to the site to have a look.

attached is the ecatc1 device0 medm, it show actual_slave_count = 290, config_slave_count = 345 which Daniel says indicates hardware error.

The Beckhoff slow controls chassis for PSL/TCS was found tripped off. Found the D13001631 Rotation Stage board for TCS Y was current limiting. Removed D6, D3, D7, and D4. Both TCS X and Y boards now have same configuration. Unit was reinstalled. PSL IO chassis will most likely need to be restarted. Past experience shows that power cycling the beckhoff chassis will cause errors with IO chassis.

h1ioppsl0 IRIG-B timing has recovered. I've cleared the IPC errors and accumulated CRC counters.

Timing error cleared @ 18:23UTC. Going to the LVEA to check on the statues of the TCS LASERs.

Actually all the front-end models are on a single computer.  Either the power for the I/O chassis has tripped, or the communications was dropped for a few cycles. If you reboot the PSL front-end computer (h1psl0), that would fix the latter.  If that fails, you need to power down the PSL front-end computer, then power-cycle the I/O chassis (resetting DC power supply if needed), then booting the PSL front-end computer.
   If that fails, then we need more investigation.

I have Jim Batch on the phone to troubleshoot

The IOP chassis was not visible from the computer, so I remotely removed the h1psl0 computer from the dolphin network and powered down the computer.  Ed power cycled the I/O chassis and powered the computer back on.  The cards in the I/O chassis are now visible.  Timing on the h1ioppsl0 model has gone "negative", meaning the DAQ data is bad until it works it's way back positive. But the PSL computer show be able to control the PSL OK.  

There is no known cause for this event.

trending h1ioppsl0's state_word shows that at 08:44 PST the FE and ADC bits faulted. It looks like a timing issue which glitched the IOP into a non-recoverable state.

IRIG-B is still in a slow recovery, DAQ data will be invalid until it completes.

Just spoke to Ed and Jonathan.  The flow rate now looks green.  It's possible that this has been triggered by other computers going down.  I can't get remote read-only MEDM access to check the status of the system.  Ed will call me if they can't key the lasers back on at the controller.

Ed and Dave got my remote access turned on so I could log in and take a look.  It appears both TCS lasers are in a good state but are not outputing power.  Power supply voltage is good.  Flow rate is good.  Temperature is good.  Current is around 0.7A indicating that they are getting RF and are ready to laze.  The Beckhoff screen shows red indicators on several read back channels, and it looks like this may be the cause of the lasers not coming on.

I've spoken to Dave and Ed again and they are going to take it from here since it looks like a Beckhoff issue. I'll be available by phone if needed.  The laser chassis will probably need to be re-keyed off and on once this has been fixed.

16:22 Also noticed that tw0 is "white". Not certain at this time how long it has been that way

h1tw0 is currently down while we are rebuilding its raid.

Actually this looks more like a conversion error between signed and unsigned words for the Fahrenheit readbacks. If you subtract 2^16/10 from these numbers, it makes sense! Filed FRS 6963.

Updated plot for the past 2 hours.  Continuing to "trend" downward.  Looks a little suspicious how the trend smoothed out around 37.