Jonathan, Erik, Dave:

At 12:28 Fri 30may2025 PDT the GC/CDS switch which provides GC networking to the MSR stopped working. Visually the switch looked functional, but it was not providing network services.

Main impact on the control room was:

 . loss of GC/CDS NAT router, no offsite access; DARM FOM stopped (unable to get references), loss of teamspeak

 . loss of all VOIP phones (they were still getting POE power, but reporting no network service).

On Jonathan's suggestion we power cycled the MSR switch. After a very long startup boot (~8 minutes) all services were recovered.

Down time was 12:28 - 12:58 PDT

DARM FOM was restarted.

This switch is called sw-osb165-msr-0. It is in rack MSR-02 U40. It is a Ruckus ICX 7850-482P (48 port, POE, 2 power supplies).

Attached photos show front and rear of the switch. To power cycle, both rear power cords need to be pulled out, orange=UPS black=FAC (note they have locking brackets which need to be disengaged). Power cords are circled in the photo.