corey.gray@LIGO.ORG - posted 06:01, Friday 28 April 2017 - last comment - 07:46, Friday 28 April 2017(35857)
12:44 Out Of Observing: EX SUS FrontEnds DOWN!!
(I'm almost cutting & pasting from what happened almost 24hrs ago! Yesterday it was EX SEI, Today it is EX SUS!)
At 12:44utc (5:44amPDT):
- Received Verbal Alarms:
- TypeError with ETMX RMS WD,
- SUS ETMX watch dog tripped,
- SUS ETMX SW watchdog tripped
- SUS TMSX SW watchdog tripped
- SUS TMSX watch dog tripped
- CDS Overview WHITE for H1IOPSUSEX, H1SUSETMX, H1SUSTMSX, SUSETMXPI
- All through this, ETMx SUS & TMS systems remained damped and H1 is still in Nominal Low Noise, but am OUT OF OBSERVING.
- Flagging this as CORRECTIVE MAINTENANCE
- Called LLO Operator (Doug L) to notify them of our status.
- Will be down on the order of hours (ie. trip to EX to restart frontends & other work)
- Will make sure NOT to touch the DITHER scripts! (thanks, for the alog to Jeff & DAY crew)
- Could not ssh into h1susex
- Richard just arrived and we'll run through what we did yesterday morning. I will let him contact Dave when he's ready.
(attached is a screenshot of all the WHITE screens we have for EX.)
Images attached to this report
Comments related to this report
Tally of activities taken to recover
- Richard (& I) had trouble logging into EX Frontends (and any other ones); this was due to the controls password being changed.
- Dave called in.
- 13:42 Richard heading out to EX
- 13:42 Since we'll probably be rebooting the EX World, taking any Guardian Nodes I can access to a SAFE state (i.e. SEI, HEPI, etc.)
- Since I did not see an obvious "SAFE" state on the SEI/HEPI/ISI Guardian Nodes (or anything in the SEI wiki), I called Hugh and he suggested I take the EX SEI Guardian Node to OFFLINE.
- 1351 h1susex booted by Richard
- 14:05 Richard back to corner station (Dave must still be working on getting us back & GREEN at EX).
- 14:11 Dave restarting models
- When we lost EX SUS, the SUS & TMS bias sliders shifted. Used Time Machine to find out values from ~5:05amPDT, but also trending Oplev signals (for etmx) & M1_DAMP pit & yaw signals (for tmsx). The sliders shifted during the reboot this morning (and yesterday morning).
- 14:35 Starting with no light on the ALSx camera---tweaking sliders to restore signals above has brought back a spot on the ALSx camera!
- (NO DITHER scripts used this morning!)
- OK, working on getting stuff to a decent state for Jim, so that hopefully all he'll have to do is an Initial Alignment.