jeffrey.kissel@LIGO.ORG - posted 12:27, Monday 22 July 2019 - last comment - 11:21, Tuesday 23 July 2019(50706)
Recovery from h1oaf1 kernel panic and subsequent Dolphin Crash Ongoing
C. Vorvick, E. Merilh, J. Kissel, J. Driggers FRS Ticket 13295 Recovery notes from h1oaf1 kernel panic (see LHO aLOG 50695) and subsequent accidental dolphin network crash (see LHO aLOG 50698): ~16:00 UTC - restored IMC optics to M1 stage OSEM valuess in PIT and YAW and PZT values to previous unlocked time we used Jul 22 2019 05:06:00 UTC - restored all corner station IFO optics (everything down stream of the IMC, including RMs, OMs, and ZMs) to just slightly after; Jul 22 2019 05:06:00 UTC. - Ran through automatic initial alignment procedure. Had to align XARM and YARM in green by hand for a bit, Ed ace'd this and got both arms up to snuff enough for the green WFS in minutes. Only other thing we had to manually intervene with: SRY alignment was not working, so Ed moved INIT_ALIGN to IDLE this keep the ALIGN IFO in ACQUIRE_SRY or DOWN. Manually misaligned SRM. Tweaked alignment of SR2 to center location on AS_C AS port QPD NOTE: We should add this occasional work-around to the automated system. - After finished with manual SR2 alignment, we accidentally requested the INIT_ALIGN guardian to go to SRC_ALIGNING which restart the entire procedure. Realized that we didn't want this a bit of the way through. So, instead, mid-stride, we forced INIT_ALIGN to jump to SRC_ALIGNING via a manual request. This worked on getting SRC back to where we wanted it, but the "false start" triggered the ALS_ARM guardians to start their initial alignment process. We stopped *that* by requesting each arm guardian to "UNLOCKED." However, this stalled the ALIGN_IFO guardian, because we'd bypassed its managerial authority. In order to prevent this in the future, Jenne Modified INIT_ALIGN such that each init align states a request ALS ARMS to go to UNLOCKED. Loaded, installed, and ARMS and ALIGN_IFO went back to happy, - Noticed that SEI_CONFIG manager guardians had stalled. By-hand requested all of the subordinates of SEI_CONF manager to be in their configuration needed for the WINDY state, and then once the subordinates were all happy (without notifications) we asked the SEI_CONF manager to INIT, and it immediately became happy. - The ETM ring heaters setpoints were wrong because they are unmonitored in the observe and wrong in safe. So, we restored to what they were 24 hours ago (a 0.3W change). Restoration occurred at ~18:20 UTC. That change means we'll be a few hours from nominal thermal conditions. (TCS C02 lasers and ITM ring heaters, were checked and confirmed OK) -Struggled with getting decent POP18 and POP90 build-up in PRMI and DRMI. Some manual intervention of alignment using PR2 (tried PR3, no dice). Noticed a slow trend upward on slow thermal timescale... wondering if, with the arms locked on green and held off resonant in red, this is ETMs off-resonant affect on sidebands (as ETMs warm up, they're doing a *better* job of *rejecting* sidebands). Just waiting, - Had to clear OM saturations from stale OMC ASC request. Didn't realize that these didn't matter for length acquisition, and graceful clear history button only (and should only) impact the OMC and OM3 history. Thought that bringing IFO to DOWN would clear it, but did not. Had to go to ASC OVERVIEW > DC CENTERING > OM1/2 P/Y CLEAR. - 19:15 UTC PRMI builds continue to increase, but DRMI is still problematic. Thinking that initial alignment of the SRC was bad, so switch to try that. ISC_LOCK to DOWN. ISC_LOCK to INITIAL ALIGNMENT. Immediately -- ALIGN_IFO to to SRC_ALIGN (before INIT_ALIGN node triggers GREEN) After SRC_OFFLOADED. ISC_LOCK to DOWN Select Resume PRMI / DRMI locked Sounds like we still need to still have a way to run individual states without starting the entire sequence. - 19:25 UTC Resume locking. More details to come as we continue recovery.
OMC guardian change:
In the past, when the OMC fails to lock on the TEM00 mode upon first try (first try is initiated before full IFO ASC is engaged), the OMC guardian would just sit at OMC_LSC_ON. Then, when we arrive at PREP_DC_READOUT_TRANSITION, if the OMC isn't ready, the main ISC guardian would make the OMC re-start it's locking sequence. However, there is no reason not to re-try locking the OMC earlier. So, now if we get to the 'stuck' part of OMC_LSC_ON, that state will now return OMC_LOCK to DOWN. Since main ISC guardian still has the request at READY_FOR_HANDOFF, the OMC_LOCK guardian then executes the DOWN state and begins to go through its whole lock sequence. Hopefully this will help ensure that the OMC is locked and ready before we get to PREP_DC_READOUT_TRANSITION, so we don't have to wait for it. Also, hopefully this will help alleviate any confusion, since previously when the OMC guardian was stuck waiting, it sent a notification up that it wasn't locked and needed help.
Strangely, when we arrived at NomLowNoise the boost and integrator for the OMC LSC servo hadn't been turned on. These showed up as SDF diffs, so they were easy to catch. I'm not sure how that would have happened though, since they are turned on in the OMC_LSC_ON state of the OMC guardian, and should have been done in order to successfully get through that state to READY_FOR_HANDOFF. But, this is the state that I had changed, so if we see these fail to come on again, I'll need to look into this. Looking at the code, I think it should be fine, so if it happens again I'm going to be confused, but I'll have a look at it then.
OMC guardian potential change:
I haven't really seen this before, but the OMC guardian got stuck on TUNE_OFFSETS. The log said that the transfer function data that guardian took were nonsensical, and suggested that one retry locking the OMC. Instead I just forced it to re-execute the state, since the measurement for the TF is taken in the main state. I'd like to change this whole state, such that the TF measurement happens in the run state (but using a counter so that it only happens once in normal circumstances), but that it can try to re-take it if it has failed. Maybe if it fails twice, then can have it return DOWN and actually redo the whole acquisition sequence. Let me know if anyone has objections, but hopefully this will be a transparent change that doesn't adverseley affect things. I'll implement it tomorrow so that I can watch it during post-maintenance acqusition to make sure it works okay.
I have added clearing the histories of the RMs and OMs to the ISC_LOCK DOWN state so that we don't run into the confusion of the OMs being unneccesarily saturated any more.