WP10232 MSR Timing Master Replacement
Daniel, Fil, Marc, Jonathan, Erik, Dave, Camilla:
I have stopped all the front end models using the 'rtcds stop --all' command. Only problem (which has been seen before) is that h1psl0 paused its stop after dbb and pmc had been stopped for about a minute before completing the stop. Presumably fss or iss paused for some reason.
I stopped the EDC with the command 'systemctl stop rts-edc_h1edc.service'
WP10192 Upgrade sw-msr-h1fe1
Jonathan, Erik, Dave:
Erik is starting the upgrade of sw-msr-h1fe1 in the MSR
WP 10232
The timing master in the MSR (Network - Timing Rack, slot U18) was replaced due to the controlled crystal oscillator control voltage out of range, alog 61673.
Unit removed S2200082
Unit installed S1103261
Summary of CDS status:
The replacement MSR timing master was in place by 09:30. The only problem was a stuck single-mode SFP in the old master, we ran without MX for about 15 minutes. Fil theninstalled a SM SFP to complete the system at 09:44.
The new sw-msr-h1fe1's SFPs did not light up. When inserted they flashed, but then no link lights were seen at either end of the fiber links. We tried port disable/enable and moving to the other fiber ports. Eventually we returned the configuration back and power cycled the switch which fixed the links.
After running for about 30 minutes, sw-msr-h1fe1 froze up, which also interrupted the network link to sw-msr-h1fe2 (DAQ switch) causing MEDMs to lose DAQ channels. It came back by itself after a couple of minutes. We are monitoring to see if this happens again.
New models for h1ioplsc0, h1omc and h1ascsqzfc were installed and the DAQ was restarted at 11:07.
As expected, the susaux system (which still use IRIG-B timing) needed an IRIG-B chassis power cycle, and each susaux computer needed to be power cycled to resync to the timing system. All susaux IOPs except for EX saw negative IRIG-B excursions which resolved in about 10 minutes.