Kept losing lock from green arms, the ALS error signals didn't look great, so I started an Initial alignment at 15:12UTC which finished at 15:33UTC. I kept losing it at ALS from XARM_WFS_DOF1_P so I turned it off (I had to do this every lock attempt) and then DIFF_IR kept reported "IR not actually found" so I had to move it around by hand. During PRMI I misaligned PRM to check on the BS and it looked great on AS_AIR, guardian took us to CHECK_MICH while I was in the middle of checking the BS.
Defiantly something is misaligned, I couldn't get PRMI for a while and finally I tried to adjust SR3 in yaw to match its OPLEV from before the glitch and that got PRMI to finally lock, but it was unstable and lost it soon after, so something else is still badly aligned, the PIT signal looked good for SR3 based on the OPLEVS. PR3 OPLEV looks a little different as well so I adjusted it back to where it was.
I have to slowly step up in states to not lose lock going through ALS and IR, even with the DOF dance Xarm just loses it for seemingly no reason. I tried requesting LOCK_SLOW_NO_WFS let it settle in LOCKING_GREEN_ARMS then requested LOCKING_ALS and I did not have to turn off any WFS for Xarm. In CHECK_IR DIFF IR was not really found as it hasn't been every attempt. As I was trying to find DIFF_IR we went to EQ mode and lost lock from the quick ground motion increase (5.0 from Mexico?). I going to go make a cup of coffee while the ground motion calms down.
Sat Oct 21 10:10:54 2023 INFO: Fill completed in 10min 50secs
TITLE: 10/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Camilla (OWL)
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 3mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.33 μm/s
QUICK SUMMARY:
15:12UTC started an IA after struggling to LOCK_ALS and seeing the ALS error signals weren't great.
IA finished at 15:33UTC, but since then I've been struggling to get DRMI, PRMI locks pretty easily but DRMI struggles even with some decent flashes, POPAIR_B_RF18 can't seem to flash above 120. Lots of ALS locklosses too, sometime from WFS sometimes for no apparent reason.
Follow up on Dolphin crash:
I looked at the logs for the front ends which had an issue at 01:05:25 PDT (lsc0, oaf0, susb123, sush2a, sush34, sush56), nothing was logged at this time.
All the IOP models experienced a very large timing glitch at this time. The size of the glitchs are
iop model | cpu max (mS) |
oaf0 | 24.3 |
lsc0 | 23.4 |
susb123 | 15.7 |
sush2a | 18.7 |
sush34 | 18.7 |
sush56 | 15.7 |
Summary: To lock green arms I put ITMs, BS, PR2 and PR3 back to Oplev positions before trip/last time we were aligning. Also touched ETMs and TMSs as usual in green arms aligning. During intal alignment I touched H1:LSC-XARM_GAIN (and reverted), PR2, PRM, SRM.
Documented in FRS 29441. Ryan C got us back to NLN at 20:16UTC by restoring all the sliders to when we were locking before the previous lock and then running another initial alignment alog 73637.
Restoring sliders with (sitemap > ASC > Init Align Compat > ! Restore) should have been the first thing to do as during the computer restart the sliders are reset to their safe.snap files, which are not (currently) recent values. Tagging OpsInfo and updated Recovery from Dolphin/Computer Crash wiki.
Investigating a dolphin crash of several corner station front ends.
I first verified for the IOPs with DACKILLs that their IO Chassis could be seen, which was the case.
I restarted all the models (to clear IOP DACKILLS) on h1oaf0, h1lsc0, h1susb123, h1sush2a, h1sush34, h1sush56. I put all the corresponding SEI IOP SWWD into bypass mode before restarting the SUS IOPs.
All the models restarted with no issues, no reboots (with their associated Dolphin fencing) were needed.
In the morning I'll see if I can find which front end caused the Dolphin crash.
The EDC is slow in reconnecting to susmc3 and susprm channels, but it looks like it will get there in time.
Sat21Oct2023
LOC TIME HOSTNAME MODEL/REBOOT
01:48:20 h1oaf0 h1iopoaf0
01:48:34 h1oaf0 h1pemcs
01:48:48 h1oaf0 h1tcscs
01:49:02 h1oaf0 h1susprocpi
01:49:16 h1oaf0 h1seiproc
01:49:30 h1oaf0 h1oaf
01:49:44 h1oaf0 h1calcs
01:49:58 h1oaf0 h1susproc
01:50:12 h1oaf0 h1calinj
01:50:26 h1oaf0 h1bos
01:52:02 h1lsc0 h1ioplsc0
01:52:16 h1lsc0 h1lsc
01:52:30 h1lsc0 h1lscaux
01:52:44 h1lsc0 h1sqz
01:52:58 h1lsc0 h1ascsqzfc
01:55:25 h1susb123 h1iopsusb123
01:55:39 h1susb123 h1susitmy
01:55:53 h1susb123 h1susbs
01:56:07 h1susb123 h1susitmx
01:56:21 h1susb123 h1susitmpi
01:58:57 h1sush2a h1iopsush2a
01:59:11 h1sush2a h1susmc1
01:59:25 h1sush2a h1susmc3
01:59:39 h1sush2a h1susprm
01:59:53 h1sush2a h1suspr3
02:01:57 h1sush34 h1iopsush34
02:02:11 h1sush34 h1susmc2
02:02:25 h1sush34 h1suspr2
02:02:39 h1sush34 h1sussr2
02:04:13 h1sush56 h1iopsush56
02:04:27 h1sush56 h1sussrm
02:04:41 h1sush56 h1sussr3
02:04:55 h1sush56 h1susifoout
02:05:09 h1sush56 h1sussqzout
After then getting zero flashes on PRMI after getting back up, I decided to try to restore all the sliders to when we were locking before the previous lock (10/19 18:19UTC) at 18:03UTC. The arms are going through increase flashes now.
Things seemed worse after the IA that I ran? During the IA everything looked good though, the signals were all reasonable and AS_AIR looked correct after each section. If this restore all doesn't help I'm going to restore things to how they were after Camilla finshed her IA earlier in the morning (~14:00UTC)
Going through each state manually and slowly, no Xarm issues this time, IR seems to be looking good too. AS_AIR looks bad at DRMI. Lockloss at CHECK_MICH
After talking with Jenne, I'm runing another intial alignment and I did not restore the optics to after Camillas IA. We're still at the 10/19 18:19UTC alignment that I restored too.
Reaquired NLN at 20:16, waiting for ADS to converge to go into observing.