david.barker@LIGO.ORG - posted 11:35, Monday 29 January 2018 - last comment - 12:02, Monday 29 January 2018(40311)
State of the CDS computing system
I'm emerging from sick leave, so this would be a good time to give a CDS computing summary:
Timing is in error. At around 3am PST Thursday 25th the MCA atomic clock time distribution system went into alarm. Trends suggest its time was varying for about an hour, and then settled to a static large offset. All three comparators in CS, EX and EY see bad times and are reporting the error.
h1oaf model is now consistently running long and reporting timing errors. This started Tue 16th Jan, presumably due to filter changes. I am investigating.
New filters need to be loaded for h1sqz, h1ascimc. If we don't get to this today, we'll cover it tomorrow during maintenance.
Surprisingly h1oaf's cpu issues are not filter related. It appears that the decommissioning of the h1ngn model on 1/16 has caused it to run long, even though there is no IPC between these models. I'll try a complete model restart on h1oaf0 tomorrow.