The interferometer had been locked and we had been observing for about 40 minutes when the TCS_ITMY_CO2 guardian knocked us out of observing. It created 3 diffs in TCSCS, and the ITMY_CO2 guardian complained it was not nominal. We couldn't get back to observing until the guardian had finished FIND_LOCK_POINT and returned to LASER_UP. Verbal has also complained several times that TCSY chill-air is low. I'm assuming for now that this is all related to the TCS work earlier.
Dave came in to talk about this. This sounds similar to this post from last month: alog#34861. This was followed by email discussion between Keita & Alastair.
Guardian had reported that the ITMY CO2 laser became unlocked at 18:09:32 PDT last night:
2017-04-05T01:09:32.92258 TCS_ITMY_CO2 [LASER_UP.run] laser unlocked. jumping to find new locking point
2017-04-05T01:09:32.98424 TCS_ITMY_CO2 JUMP target: FIND_LOCK_POINT
So the SDF differences raised and the fact they terminated observation mode appears to be correct.
Jeff K cleared up the confusion of what should and shouldn't be monitored in this case.
The filter modules in question should not be monitored, they are being changed by guardian during observation. The TSTEP channel records the GPS time a step is made, and should never be monitored.
Taking TSTEP as an example, I checked through the SVN repository at the observe.snap file for TCS CO2 ITMX and ITMY and found that in Oct 2016 both were not monitored. In 3rd March this year they both were monitored. On 22 March ITMX was not monitored but ITMY was. We suspect that by accident too many changes are being applied to the snap files, for example perhaps monitor-all was applied.