TITLE: 05/26 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG:
9:35 Begin resetting Watchdogs -
- OFI WD is tripped. I can't find how to un-trip this. It isn't accessible from the Ops Overview screen.
- Found it! SITEMAP>SUS_AS.
9:57 BLRMS below 1um/s - AS spot is low and right
- Bringing SEI_CONFIG back to WINDY
10:07 Everything un-tripped. Everything re-aligned - AS Spot still low and right
- no flashing on ALSY camera
- reset BS and all Y optics back to witness sensor values prior to EQ
- Oplev spots look good
11:40 Just got off the phone with Sheila (she called in to check on the quads). I didn't push the big green RECOVER EQ button!!!!!! -_-
- Things look a LOT better now.
11:45 Begin Initial Alignment
12:10 Raine from LLO called to let me know that ALSY tripped and requires a trip to the end station - phone buddy system in effect
12:33 Raine reported back at the corner station.
12:52 begin re-locking H1
14:00 Lock failed at INCREASE POWER
14:23 Lock failed at INCREASE POWER (again)
- Going to wait at ENGAGE_DC_VIOLINS
- ETMY MODES 1 & 6 are rung up pretty good - gonna take a while
15:00 Handing off to Cheryl
Cheryl, Sheila
Here are links for the two locklosses from increase power that Ed logged: 1242914440 and 1242915786
In both cases the DSOFT P and CSOFT P control signals became large right after we entered the INCREASE POWER (504) state. About a week ago we were having trouble suppressing the radiation pressure instability in the X arm because the beams were falling off the QPD. As a temporary solution we introduced an offset on TMSX TEST P before we increase the power, which is turned off again for lock acquisition so that we do not loose our green alignment references. We set the ramp time on the offset to 30 seconds, but this must have been reset to 2 seconds this morning either by the suspension guardian when recovering from a watchdog trip or by the dither alignment script.
We set the ramp time to 30 seconds again, and I added a line to the guardian that sets the ramp time to 30 seconds before turning on the offset in INCREASE_POWER. Cheryl loaded this and was able to get through the first few steps of the power increase.
Both the lockloss tool and the new remote access made it much easier than it might have been to diagnose and solve this issue remotely than it would have been in O2.
So we don't think this was simply a symptom of ETMY Modes 1 and 6 being rung up after a very large seismic event? I can say that I hadn't noticed them until the third attempt.