WP8697, FRS11690

Richard, Fil, Dave:

Summary: Replacing the last (5th) 18bit DAC with a 20bit DAC fixed h1susey's booting problem. The removed DAC will be tested offline to verify it has failed and was the cause of the problem.

Details:

Following yesterday's model starts causing a lockup of the computer, Fil went onto site to power cycle the IO Chassis and replace cards if needed. We decided to take this opportunity to replace the last 18bit DAC with its 20bit DAC upgrade for the ESD drive (as was done some time ago for h1susex, covered with FRS11690).

1: performed a complete power cycle of CPU + IO-Chassis, computer will not boot.

2: powered down computer, pulled one-stop fiber connection and the powered it up. Computer booted, problem with IO-Chassis

3: As part of removing the five 18bit-DAC cards, one at a time, we started with the fifth card in slot 2-5 and replaced it with its 20bit DAC upgrade. Computer booted, perhaps we got lucky on the first attempt.

4: But... the IOP model will not start. DMESG reports Dolphin IX driver startup problems, with the /etc/dis/dishosts.conf file missing.

I verified this file was actually missing on h1susey. /etc/dis is a sym-link to the local RAM-DISK /var/log. During the boot process several dis files are copied into this location but dishosts.conf is not.

5: on h1boot1, I restarted the Dolphin IX network manager. This fixed the problem, when h1susey was next booted its dishosts.conf file was created in /var/log and the models started correctly.

I suspect the dis_networkmgr process had been running since the last boot of h1boot1, 292 days. Note that we did not have this issue when booting h1sush34 on 8th June, so the problem has appeared since then.

6: Jim and Cheryl completed the startup of the SUS and SEI models.

Before we installed the 20bit DAC, I made a change to the h1iopsusey.mdl model to replace the 5th 18bit DAC with its 20bit DAC replacement. After the reboots this model is now running.

Note that because the 20bit DAC change has not yet been made in the h1susetmy.mdl model, no user model is using the new DAC and the IOP model is maintaining its DACKILL status (a safety system whereby if zero channels of a DAC are being driven, the IOP forces all of its channels to be zero and turns off the keepalive signal to the AI-chassis, thereby disabling its outputs).

This is the reason there is a DACKILL RED LED on the overview for h1iopsusey. This will be greened when h1susetmy takes ownership of the new DAC.