With the Dolphin Crash at around 8pmPT, for H1, it primarily affected the Corner Station's SUS & SEI (but not every chamber). After Dave restored the CDS side of things, we worked on bringing back the SUS and then the SEI (with Rahul and Jim). Some notes:
- PSL appeared to not be affected by the Dolphin Crash (with RefCav & PMC remaining locked)
- The IMC locked (although it was with a bright IMC REFL spot)
- Various Corner Station chambers appeared to not be affected by the crash (i.e. HAM1, HAM7, HAM8)
- Started a bit positive and thought I could jump right into an Initial Alignment, but there was no green light for x & y.
- At this point, I looked at pit/yaw sliders for optics and worked on restoring them.
- Since not ALL SUS were affected by the Dolphin Crash I went one by one restoring suspensions. I used Time Machine and also did individual Restores of optics via the IFO COMPACT ALIGN medm.
- In hindsight, I should have restored ALL optic sliders vs going one by one.
- Sliders restored to ~01:00utc (during the lock from earlier in the shift & before the Dolphin Crash).
ALIGNMENT #1:
- GREEN X-Arm: This took several iterations/pauses, mainly because Green X-arm was low. I restored sliders to where they were when locked, and touched up ETMx & TMSx the best I could with UNLOCKED flashes, but only could manage about flashes up to 0.6 (which is not high enough to trigger a lock for the X-arm). I even tried tweaking ITMx a little. But since I was getting nowhere, I backed out everything and eventually restrored ITMx/ETMx/TMSx back to their slider values when locked.
- This seemed to help, as I finally had flashing above 0.6.
- I then let INCREASE FLASHES take over, and it eventually locked Green X-arm.
- Along the way, I feel like I caused issues with this alignment because I would interrupt INCREASE FLASHES so that I could touch up alignment while UNLOCKED. I also ran the DOWN script with ALIGN_IFO at various points here. Which might have complicated things....
INIT_ALIGN Errors:
- The INIT_ALIGN guardian ended up going into ERROR several times during the night for me.
INPUT ALIGN:
- X-arm would not lock, and since the Alignment checklist says to adjust PR2, I saw that its sliders were off. Once they were taken to a good time Xarm would show flashes and eventually lock (but it was also touchy). If Xarm would break lock, it would take down the IMC. However, the x-arm did stay locked enough to complete INPUT ALIGN.
- See below for the errors in the log.
Once INPUT ALIGN was completed, I continued with ALIGN_IFO to complet the Initial Alignment (I did not use INIT_ALIGN because it wasn't working for me---due to the ERRORs).
LOCKING:
- Green arms locked up fine.
- FIND IR mostly went fine.
- DRMI locked fast, but quite a bit of movement was seen on AS_AIR
- SRM Saturations were on Verbal the whole time when DRMI locked.
- There was a lockloss a few steps after DRMI.
- And both HAM4 & HAM5 ISIs tripped at the lockloss.
Since I was worried the alignment was not run correctly due to the INIT ALIGN error, I decided to run another alignment letting ALIGN IFO use INIT ALIGN.
ALIGNMENT #2:
Mostly a straightforward alignment, BUT for SRC ALIGN: There were SRM saturations via Verbal---not a good a sign. These saturations would continue even after offloading. So something had to be amiss---I must have missed something from the crash.
Since it was almost midinght, I went for another lock, and sadly, H1 made it through DRMI, but it continued to have SRM saturations and at the lockloss the HAM4 & HAM5 ISIs tripped again. This is when I handed off to Ryan.
Additional NOTE:
- I have not looked at IMC PZTs at all, because I assumed that since the PSL seemed minimally affected and since the MC REFL spot was improved after MC1/2/3 sliders were restored---I just assumed the PZTs were fine. But in addition to the noted SRM saturations, there have also been MC2 saturations.
Here Is the INIT_ALIGN error:
2023-05-23_06:26:02.156247Z INIT_ALIGN [LOCKING_GREEN_ARMS.main] timer['y_locked_but_no_wfs'] = 0
2023-05-23_06:27:02.154391Z INIT_ALIGN [LOCKING_GREEN_ARMS.run] timer['x_not_locked'] done
2023-05-23_06:27:02.156131Z INIT_ALIGN [LOCKING_GREEN_ARMS.run] timer['y_not_locked'] done
2023-05-23_06:28:41.334397Z INIT_ALIGN W: Traceback (most recent call last):
2023-05-23_06:28:41.334397Z File "/usr/lib/python3/dist-packages/guardian/worker.py", line 494, in run
2023-05-23_06:28:41.334397Z retval = statefunc()
2023-05-23_06:28:41.334397Z File "/usr/lib/python3/dist-packages/guardian/state.py", line 246, in __call__
2023-05-23_06:28:41.334397Z main_return = self.func.__call__(state_obj, *args, **kwargs)
2023-05-23_06:28:41.334397Z File "/usr/lib/python3/dist-packages/guardian/state.py", line 246, in __call__
2023-05-23_06:28:41.334397Z main_return = self.func.__call__(state_obj, *args, **kwargs)
2023-05-23_06:28:41.334397Z File "/opt/rtcds/userapps/release/isc/h1/guardian/INIT_ALIGN.py", line 130, in run
2023-05-23_06:28:41.334397Z nodes[f'ALS_{xy}ARM'] = 'INITIAL_ALIGNMENT_OFFLOADED'
2023-05-23_06:28:41.334397Z NameError: name 'xy' is not defined
2023-05-23_06:28:41.335859Z INIT_ALIGN [LOCKING_GREEN_ARMS.run] USERMSG 0: USER CODE ERROR (see log)
2023-05-23_06:28:41.388535Z INIT_ALIGN ERROR in state LOCKING_GREEN_ARMS: see log for more info (LOAD to reset)
2023-05-23_06:28:54.822757Z INIT_ALIGN LOAD REQUEST
2023-05-23_06:28:54.827322Z INIT_ALIGN RELOAD requested. reloading system data...
2023-05-23_06:28:54.872241Z INIT_ALIGN module path: /opt/rtcds/userapps/release/isc/h1/guardian/INIT_ALIGN.py
2023-05-23_06:28:54.872241Z INIT_ALIGN user code: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_library.py
2023-05-23_06:28:54.872552Z INIT_ALIGN user code: /opt/rtcds/userapps/release/isc/h1/guardian/lscparams.py
2023-05-23_06:28:54.872552Z INIT_ALIGN user code: /opt/rtcds/userapps/release/sys/h1/guardian/timeout_utils.py
2023-05-23_06:28:55.722360Z INIT_ALIGN RELOAD complete
2023-05-23_06:28:55.723769Z INIT_ALIGN W: RELOADING @ LOCKING_GREEN_ARMS.run