WP 7101
Sheila, Richard, Fil, Sudarshan, Dave:
We power cycled the front end computers and their associated IO Chassis for the systems h1susb123 (ITMX, ITMY, BS, ITMPI), h1susex (ETMX, TMSX, ETMXPI) and h1susey (ETMY, TMSY, ETMYPI). Prior to the reboots, Sheila checked the SUS safe.snap SDF files to see if they were up to date (which they were).
The power down sequence for each computer was:
- stop running models
- take computer out of Dolphin fabric
- remotely power down the computer
- power down the IO Chassis via front end rear switches
The power up sequence was:
- Power up the IO Chassis (rear and front switches)
- power up the computer (front panel switch)
The power sequence in the corner station went well. We had problems at both end stations:
EX: the power up of h1susex caused the h1iscex computer to freeze, which in turn caused a Dolphin glitch on h1seiex.
EY: the power up of h1susey caused a dolphin glitch on this fabric, all ISC and SEI models were glitched.
Both problems were unexpected and unexplained and worrisome.
h1iscex was found to be frozen but powered on. Richard power cycled the computer.
The recovery from the Dolphin glitches at both end stations was the same:
- stop all models on ISC machine, stop all models on SEI machine
- start SEI models
- start ISC models
note, h1iopseiey had a slight IRIG-B excursion to +50, which recovered in a few minutes.
Once all the models were running correctly, the system was cleaned up by resetting the IOP software watchdogs (SWWD), clearing the latched errors with DIAG_RESET, clearing the DAQ CRC errors.
Sudarshan reports a PCAL guardian issue with HIGH_FREQ_LINES node, which did not like h1calex being reset to its safe.snap settings.
While we were rebooting h1susey, Richard and I took a look at the BIOS settings on this computer (one of the faster models). We found that the 'Power Technology' setting is set to 'Max Performance', which Gerrit reports could be the source of our glitching.