Reports until 11:47, Friday 14 December 2018
H1 SEI (Lockloss, OpsInfo)
jeffrey.kissel@LIGO.ORG - posted 11:47, Friday 14 December 2018 (45938)
ITMY STS Problems and Thus Sensor Correction Problems -- Just a missing SDF Setting / Cal Filter from new SEIPROC model
J. Driggers, J. Kissel, R. McCarthy, T. Shaffer, P. Thomas, N. Lecoeuche

During last night's front end crash (LHO aLOG 45925), the new SEIPROC model (sLHO aLOGs 45641, 44927, and 44906) on the h1oaf0 machine also crashed.

Sadly, to-date, the SEIPROC model, because it's so new, had not been added to the SDF overview screen, and thus it took a while to discover that the calibration filters were OFF on all corner station GND STS2s.

This was discovered and solved via a very indirect way:
 - Jenne & Niko found the Y-arm would not stay locked in green, and they also noticed that the DMT Seismic trace for the EQ and Microseism bands on the wall FOM had droped by a factor of 10-20, only on the "CS Z axis" trace. 

The solutions (sadly) turned out to be much more drastic than were needed:
 (0) Jim and Hugh -- the folks whom are the primary experts on the sensor correction construction zone are off site today.
 (1) Those listed above (here-to-forth referred to as "we") restarted the DMT monitor, thinking something might have gone wrong the the DMT process. This didn't fix the problem.
 (2) We turned off "all" *corner station* sensor correction -- but did so using the individual chamber configuration managers, not the overall site config manager. We found out later, this does *not* turn off HEPI Z sensor correction (only the overall site manager does). Thus, "turning off the sensor correct" didn't fix the problem.
 (3) Unfortunately, given the sensor correction construction zone from the new SEIPROC model converting models to be able to use Common Mode sensor correction 
    (a) it's difficult to find what sensor correction is on or off (some links are broken, the "final answer" channel that receives the given faded STS signal is small and tricky to find, and how to turn it on/off manually is now a bit confusing)
    (b) we've lost the ability to switch between corner station STS2s, so the ITMY STS2 in the corner is now the only option; we can no longer use an input matrix to switch between ground sensors.
    (c) (repeating the above message) the SEIPROC model was not yet on the SDF overview screen, and we didn't yet think to go to the CDS overview screen and open up the SDF table from the GDS_TP screen
 (4) Beginning to suspect (very errantly) that "the world" going down last night meant that somehow the ITMY STS had "gone faulty," -- with *a* clue being that an ASD of all three corner station STS channels showed that the Y channel on ITMY STS was different than other channels -- we "double confirmed" (as best we could; see (3a)) that sensor correction was OFF, I power cycled the STS interface chassis for ITMY.
    Sadly, because of (2), this sent a terrible signal to all HEPI Z sensor correction that was *still on*, and tripped all platforms in the corner station.
 (5) Even worse (because the analog STS wasn't the real problem), we saw that this did not change the odd character of ITMY STS. Thus, before untripping platforms, I went back to the STS interface chassis and *tried* to flip the mode of the STS to a 1-sec period, and auto-zero the instrument. Confusingly, these interface chassis switched had *no* effect on the instrument. Defeated, I punted on analog investigations of the STS.
 (6) Miraculously, Jenne finally found the location of the calibration filters under sitemap > SEI > "ITMY GND STS," finding the FM1 "Cal" filters were OFF, and 
    (a) TimeMachine said it was on yesterday, and
    (b) The filter was a factor of ~10, which is what were were missing from the DMT wall FOM.
  So we turned all the CAL filters back on.
  Problem solved!

We then recovered from our self-induced destruction and
 (i) Untripped and re-isolated all corner station platforms, which recovered the Y-arm green locking, and
 (ii) added the SEIPROC model to the SDF screen, and commited to the userapps repo -- discovering that these filter banks / channels weren't even *initialized* let along monitored, so even if we'd found the SDF table, it would have taken an extra layer of expertize to see that they were OFF.


 

Images attached to this report