WP12186
Richard, Fil, Erik, Dave:
We performed a complete power cycle of the h1psl0. Note, this is not on the Dolphin fabric so no fencing was needed. Procedure was
- Stop all models and power down h1psl0 computer
- Power down IO Chassis, all AA chassis and all AI chassis.
- Wait for a minute
- Power up AA chassis and IO Chassis (keep AI's off)
- Power up h1psl0 computer, wait for models to start
- Finally, power up AI chassis.
The system was power cycled at 10:11 PDT. When the iop model started, it reported a timing error. The duotone signal (ADC0_31) was a flat line signal of about 8000 counts with a noise of a few counts.
Erik thought the timing card had not powered up correctly, so we did a second round of power cycles at 10:30 and this time the duotone was correct.
NOTE: the second ADC failed its AUTOCAL on both restarts. This is the PSL FSS ADC.
If we continue to have FSS issues, the next step is to replace the h1pslfss model's ADC and 16bit DAC cards.
[ 45.517590] h1ioppsl0: INFO - GSC_16AI64SSA : devNum 0 : Took 181 ms : ADC AUTOCAL PASS
[ 45.705599] h1ioppsl0: ERROR - GSC_16AI64SSA : devNum 1 : Took 181 ms : ADC AUTOCAL FAIL
[ 45.889643] h1ioppsl0: INFO - GSC_16AI64SSA : devNum 2 : Took 181 ms : ADC AUTOCAL PASS
[ 46.076046] h1ioppsl0: INFO - GSC_16AI64SSA : devNum 3 : Took 181 ms : ADC AUTOCAL PASS
We decided to go ahead and replace h1pslfss model's ADC and DAC card. The ADC because of the continuous autocal fail, the DAC to replace an aging card which might be glitching.
11:30 Powered system down, replace second ADC and second DAC cards (see IO Chassis drawing attached).
When the system was powered up we had good news and bad news. The good news, ADC1 autocal passed after the previous card had been continually failing since at least Nov 2023. The bad news, we once again did not have a duotone signal in ADC0_31 channel. Again it was a DC signal, with amplitude 8115+/-5 counts.
11:50 Powered down for a 4th time today, replaced timing card and ADC0's interface card (see drawing)
12:15 powered the system back up, this time everything looks good. ADC1 AUTOCAL passed again. Duotone looks correct.
Note that the new timing card duotone crossing time is 7.1uS, and the old card had a crossing of 7.6uS
Here is a summary of the four power cycles of h1psl0 we did today:
Card Serial Numbers
Detailed timeline:
Mon04Nov2024
LOC TIME HOSTNAME MODEL/REBOOT
10:20:14 h1psl0 ***REBOOT***
10:21:15 h1psl0 h1ioppsl0
10:21:28 h1psl0 h1psliss
10:21:41 h1psl0 h1pslfss
10:21:54 h1psl0 h1pslpmc
10:22:07 h1psl0 h1psldbb
10:33:20 h1psl0 ***REBOOT***
10:34:21 h1psl0 h1ioppsl0
10:34:34 h1psl0 h1psliss
10:34:47 h1psl0 h1pslfss
10:35:00 h1psl0 h1pslpmc
10:35:13 h1psl0 h1psldbb
11:43:20 h1psl0 ***REBOOT***
11:44:21 h1psl0 h1ioppsl0
11:44:34 h1psl0 h1psliss
11:44:47 h1psl0 h1pslfss
11:45:00 h1psl0 h1pslpmc
11:45:13 h1psl0 h1psldbb
12:15:47 h1psl0 ***REBOOT***
12:16:48 h1psl0 h1ioppsl0
12:17:01 h1psl0 h1psliss
12:17:14 h1psl0 h1pslfss
12:17:27 h1psl0 h1pslpmc
12:17:40 h1psl0 h1psldbb