Another event at 14:02 PST , proc status file for IOP model posted below.

If this is a timing error (and at this point we are not certain of this), our first change was to replace the fiber optics cable between the IO Chassis timing slave and the timing fanout chassis. A new cable was installed before the system was restarted.

Note, the existing fiber was found to be not cable tied to the bundle of fibers at the timing fan-out card, rather was floating by itself.

Sequence was:

• kill all models in correct sequence
• take h1oaf0 out of the dolphin fabric
• power down h1loaf0 computer
• power down IO Chassis
• replace timing fiber
• power up IO Chassis *
• power up h1oaf0 computer
• wait for irig-b to come back to positive values (approx 3 mins)

* we did not do the TCS-AI power sequence this time and unfortunately this may have tripped the TCS chillers.

controls@h1oaf0 ~ 130$ cat /proc/h1iopoaf0/status
startGpsTime=1162838354
uptime=12257
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162838680
adcHoldTime=16
adcHoldTimeEverMax=91
adcHoldTimeEverMaxWhen=1162850544
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=3
cycle=49476
gps=1162850611
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,5
cpuTimeMaxCycle(cur,past sec)=21,21
cycleHist: 3=65183@17 4=350@65535 5=3@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=91 Current=14
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0

press DIAG_RESET

controls@h1oaf0 ~ 0$ cat /proc/h1iopoaf0/status
startGpsTime=1162838354
uptime=12333
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162838680
adcHoldTime=14
adcHoldTimeEverMax=91
adcHoldTimeEverMaxWhen=1162850544
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=4
cycle=43808
gps=1162850687
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,6
cpuTimeMaxCycle(cur,past sec)=21,1
cycleHist: 3=65201@17 4=333@65535 5=1@0 6=1@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=17 Current=15
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0