corey.gray@LIGO.ORG - posted 07:21, Friday 13 March 2020 - last comment - 17:07, Tuesday 31 March 2020(55578)
H1 Range Drop, Lockloss, and then Back to OBSERVING
Summary: At about 8:30utc, H1 took a 20Mpc drop in range & hovered there for ~4hrs until it dropped out of lock. Cursory look didn't show anything obvious. H1 was able to get back to NLN without intervention in under an hour and the range appears to be normal-ish so far.
Timeline:
- 8:37utc (1:37amPT) H1 range begins to drop over a few minutes from ~119Mpc down to 102Mpc (no notification received, other than Richard text at 13:03), and stays here until...
- 12:24 LOCKLOSS
- 13:19 NLN
- 13:22 OBSERVING...range is hovering around 116.5Mpc (after first few points around 119Mpc)
Quick Check:
Did a quick check on a few items to see about reasons why H1 took the drop to 102Mpc, and looked at "usual suspects":
- DIAG_MAIN: NOTHING observed in the log.
- SQZ_MGR: NOTHING observed in the log.
- ISC_LOCK: NOTHING observed in the log.
- SUS Violins: NOTHING observed rung up above 3 for fundamental and 1st harmonic.
It looks like the issue was ETMY violin mode 2 (505.0794hz) started a very slow ringup until the DCPDs started to saturate and we began to lose range. This mode normally has a gain applied but the violin Guardian turned it off yesterday, March 12 00:49 UTC. The mode didn't increase for about 8 hours, but then started a slow increase around Mach 12 08:30 UTC. Why it changed and started to increase at this is still under investigation.
The irony here is that there was plan for Rahul to change the violin Gaurdian today to handle this exact situation. A day too late unfortunately.
Adding a better plot of when the DCPDs saturated.
I am attaching the ndscope plot for ETMY mode 2 from March 13 - which resulted in DCPD lockloss. At the lockloss time the monitor level was around 6, if we go back 2 hrs in time then the monitor level was around 5.5. If we choose this value (monitor level 5.5) as a threshold for setting up a verbal alarm and IFO_NOTIFY, then operators will have 2 hours of advance warning, giving them some time to damp the mode manually.