david.barker@LIGO.ORG - posted 16:27, Monday 11 August 2014 - last comment - 16:58, Monday 11 August 2014(13341)
Summary of recent rash of CDS problems
Jim, Dave
With so much going on this weekend and today, I thought it would be nice to provide a top level summary of events.
The LDAS raid issue is thought to be completely unrelated to the various FE issues, and was purely coincidental.
There is no obvious connection between the various FE issues, but we cannot imaging they are coincidental.
All times are PDT.
LDAS disk9 failure in RAID
| Fri 8/8 21:57 | Warning message disk9 |
| Fri 8/8 22:37 | Error Disk9, continuous |
| Sat 8/9 01:52 | h1fw0 single restart |
| Sat 8/9 04:02 | h1fw0 regularly restarting |
| Sat 8/9 15:57 | problem resolved, disk9 removed from raid |
h1sush2a CPU freeze
| Sat 8/9 00:06 | cpu freeze, DAQ data bad for many other FE |
| Sat 8/9 11:48 | pwr CPU, large IRIGB error on IOP |
| Sat 8/9 11:54 | restart IOP, still large IRIGB errors |
| Sat 8/9 12:11 | pwr CPU and IOChassis, all is good |
h1seib1 CPU freeze
| Sun 8/10 18:46 | cpu freeze, DAQ data on others good |
| Mon 8/11 08:35 | pwr cpu , IRIGB drifted bad |
| Mon 8/11 08:39 | IRIGB drifted good, all is good |
h1susb123 CPU freeze
| Mon 8/11 13:46 | cpu freeze, DAQ data good on other FE. SWWD causes SEI trip 5 mins later |
| Mon 8/11 13:58 | pwr cpu. All looks good but DAC is undrivable (discovered later) |
| Mon 8/11 14:34 | DAC problem discovered, fixed with restart of IOP |
h1seih23 DACs not driving
| Sat 8/9 15:30 | h1seih23 DACs not being driven |
| Mon 8/11 10:42 | fixed by restarting IOP |
here are the console images for the three cpu freeze up events (h1sush2b, h1seib1, h1susb123)