CDS Power-outage-recovery status

We are still recovering CDS from Thursday's power outage. All critical systems have been recovered and IFO locking was started yesterday.

Both Alarms and Alerts systems are operational and sending cell phone texts, I'm having issues sending alert emails.

Here is a brief summary of we are currently working on:

Timing:

Timing has two issues. The main one is with EY timing fanout chassis, its single mode link to MY (port15) is showing delay issues. Despite this h1pemmy appears to have good timing.

The secondary issue is that the atomic clock in the MSR has time jumped by 0.4S and needs resyncing.

Disk Failures:

The MSR file cluster has lost 4 disks. Resilvering for 2 of them is ongoing.

h1susauxh2 Power Supply Failure:

One of h1susauxh2's power supplies has failed, FRS36264.

EDC Disconnected Channels:

EDC currently has 574 disconnected channels, relating to auxiliary IOCs which still need to be started.

CDS OVERVIEW

Referencing the CDS Overview attached:

Range 7-segment LED service needs to be restarted.

Range of -1 MPc shows an outstanding GDS issue

EDC disconnect count mentioned above

Timing RED because of issues covered above

Picket Fence WHITE, needs restarting (see list below)

CDS Alarm RED due to EDC disconnect count

CDS SDF YELLOW because of remote power control and FCES-WIFI issues

Missing Auxillary EPICS IOCs

List may not be complete:

Picket Fence, End Station HWS, Mains Power (CS, EX), SUS Violin monitor, ncalx, cds load mon h1vmboot1, cal inj exttrig, range led, Observation mode