Summary

On Saturday 04 May 2019 there was an alignment problem for Y arm starting at Corey's owl shift (alog 48973) throughout Ed's shift alog 48979) and into Patrick's evening shift (alog 48982).

Seems like a part of the problem was a big temperature swing over multiple days, a problem first reported by  Bubba (alog 48936).

Symptom

In the first attachment that is about 20 hours' worth of trend, t=0 is when the lock was lost during Corey's shift.

ETMY M0 LOCK_P was absolutely huge just before the lock loss, -400ct ~ -20urad. As a result, after the lock was lost, ETMY moved by about +17 urad in PIT (my scribbling in yellow).  There were also some non-negligible jumps in ITMY and TMSY, but nothing comparable to ETMY.

After an hour and a half or so Corey managed to lock ALSY with OK alignment twice (cyan scribble), and then couldn't lock for a while and again manages to lock twice (red scribble), but nothing came back to where they should be.

Temperature VS drift of suspension(s)

Second attachment is the same as the first one except for the time axis which is from about 10 days ago until now. t=0 is still Corey's lock loss.

The most revealing is the correlation between reaction chain top mass P and the temperature, because reaction chain M0 is not affected by ASC at this time scale as ASC feedback to ETM goes to test mass chain M0 at DC via an integrator.  I didn't show it here but the temperature change of the EY VEA (as opposed to PEM BSC10 temperature sensor) was almost 2F peak to peak.

The drifts are huge, more than 20urad for ERM top mass, TMSY is a bit harder to tell due to initial alignments but seems to be ~20urad too.  As a result, during the long lock ASC had to push ETMY really hard to keep it aligned, but when the lock was lost the feedback is zeroed and the ETMY experienced a huge kick.

During recovery, the suspensions were still moving fairly rapidly. L2 witness sensors would have been useless for long term trend because ERM has drifted and was drifting, which means that we didn't have a reasonable long term reference without knowing that. The drift only slowed down after the first NLN lock was acquired by Patrick (red vertical line that I've overlayed on the plots).

In conclusion, it seems extremely likely that ETMY and TMSY drifted by a huge amount due to the temperature swing in EY VEA, causing a sudden huge kick when the lock was lost. Recovery was difficult because things were still moving as we were working, and our usual long term references, i.e.  L2 witness sensors, were unreliable due to ERM chain drift.