corey.gray@LIGO.ORG - posted 03:09, Wednesday 08 May 2019 - last comment - 03:09, Wednesday 08 May 2019(49093)
Multi-System Failure for H1! (edit: Dolphin Network failure for LVEA)
During the shift change, we had multiple things fail on us. While I can remember going to list what we saw/noticed in order:
- 7:04:05 "TCS[x & y] chiller trip imminent" on Verbal Alarm
- 7:04:17 All LVEA ISIs tripped, HAM236 HEPIs tripped
- 7:04:49 BS ITMy HEPIs tripped
- Looked up and we saw lots of RED on CDS overview for LVEA (& a few bits for EX & EY)
- Ran out to check TCS chillers (I knew these needed their weekly check, so wanted to check)
- 7:08 Contacted LLO & Virgo via TeamSpeak
- 7:10 Called Keita, and he asked me to call Dave
- 7:15 Called Dave
- gave him rundown of what we saw here, and he was going to check things
- Took OBSERVATORY MODE to CORRECTIVE MAINTENANCE
- Screenshot of main windows we saw.
Images attached to this report
Comments related to this report
7:27 Beginning Dolphin recovery (Dave).
- Talked with Dave and he said this is a Dolphin failure for the LVEA (except the PSL) with IPC errors for EX & EY
- 7:27 Dave started models in this order: SUS, SEI, ISC/LSC, OAF1
- Model restarts would take systems to their save.snaps
- Dave asked me to:
- untrip Software Watchdogs
- untrip SUS & SEI
- Dave handed over to me and said I can start recovery & we can go from there.
- If Dolphin network crashes again we will wait until morning (not call anyone).
- ...recoverying all the watchdogs!
- Still see lots of message on DIAG_MAIN (see screenshot from initial entry)
- Reset all Software Watch Dogs.
- Now moving on to Untripping SUS & SEI.
- Order I will take are HAMs and then BSCs
- For each chamber I will start with the (1) payloads (i.e. SUS), (2) HEPI, & (3) ISI
NOTEs:
- Dave might not have received text about this, because I asked him when I called (but he said he was going to check).
- Ops Overview: OFI (SUS) at HAM5 button doesn't work
- 8:09 Completed untripping SUS & SEI watchdogs
IMC was not locking.
- Ran DOWN for IMC_LOCK
- then it locked when I requested ACQUIRE.
- set to LOCKED state with no issue.
- 8:19 IMC restored
Moved on to Locking
- Locking Arms Green
- 8:17 No spot for X & ugly spot for Y
- 8:21 ALSx came back all the way on its own! ALSx restored (Y-arm no change)
- 8:40 Y arm restored
- During the model restart looks like ITMy's save.snap took the pit & yaw offsets to a different spot (see attached)
- Restored ITMy pit/yaw offsets & now had flashes!
- UNLOCKED and tweaked up flashes via ETMy
- ALSy restored
- Moving on to rest of locking & will not keep a log (unless I begin to have issues at later locking steps.
- At this point we appear to be restored from the Dolphin crash (atleast from an operator's perspective).
Images attached to this comment
