ryan.short@LIGO.ORG - posted 18:03, Sunday 30 April 2023 - last comment - 02:57, Monday 01 May 2023(69194)
Recovery Following CS Front Ends
Following the CS front end crash and recovery effort by team CDS, I have recovered all suspensions and remaining seismic systems. I noticed the alignment on the AS AIR camera looked poor, so I decided to restore all optics to their alignments at 14:17 UTC on 4/29 (the last time we locked ALS). As I'm not available to babysit H1 this evening, I've told it to relock itself and will leave it be. This should be a useful test of the automation as the TRX/Y signals are floating around 0.2; not amazing flashes to start with.
I glanced at the control room screenshots around 6:45, and saw that the IFO was stuck because it could not find the COMM IR.
I helped it out, but after looking at the ALS Camera error signals (and the poor DRMI flashes) we need an initial alignment, which I will start now.
Initial alignment mostly worked by itself. The BS was enough out of alignment (which, actually could have been due to me poking the sliders when I was trying to lock before aligning) that it couldn't lock MICH_BRIGHT. It was doing a nice job of going to DOWN in ALIGN_IFO though. While it was running the DOWN of ALIGN_IFO, I poked the BS sliders to get better flashes on the AS_air camera, and then it caught MICH and kept going.
After that, initial alignment went on all by itself and completed successfully.
The flashes look better. I'm going to let the automation take it from here (we just started to try DRMI), and may check back in a while. Others are free to take a look though, if it looks like it needs more help.
Once in the last few hours we made it to Engage_ASC_for_full_IFO, after I poked the SRM and SR2 to convince DRMI to lock (after a few PRMI aligns). But, the buildups looked very ratty and poor.
Just now, I noticed a notification that SEI_CS was not at its nominal configuration. The notification had a helpful note to re-request the state, so I clicked Init on SEI_CS, and it seems to have re-executed whatever it needed, to get back to Config_windy. The notification is now gone.
I wonder if, when the computers came back up, the guardian didn't know that it should do something to reset the seismic config - if that's what happened, we should think about what we can do to improve this in the future, as we work toward more automation.
I'll check back in on the IFO if I wake up in the middle of the night, but I'm hopeful that things will be more smooth now.
SDFs accepted, with some names in bold for folks to have a look and double check me:
* SUSETMY, 1st attachment. Same as accepted by AustinC in alog 69176. I'm unsure why they needed accepting, since it looks like RyanC and Austin had both recently accepted them. But, it's probably becasue this is one of the models that previously was linked between the safe and observe, and now is not. Hopefully this won't come up agian.
* SUSPROC, 2nd attachemnt. These are filters for violin mode monitoring (not damping), so I've reverted them to match how they were on Friday (alog 69129). Rahul, please have a look, to see which way they should actually be, and check their safe.snap so that they come up after reboot with the filters that are correct.
* CALINJ, 3rd attachment. It looks like there was a calibration switch that was off, but had been on for the last 2 weeks. Before turning it back on (since it's the kind of thing that will send real siganl to the IFO), I went to NLN_CAL_MEAS so that any ramping would be taken care of nicely. I then reverted the SDF, and requested Nom_Low_Noise. After that, I have an error in the CAL_AWG_LINES guardian (4th attachment). I hit the Load button, and it was happy. I have not looked at the data, but this may mean that I reverted and turned that switch on while some calibration lines were still going. Seems to not have caused an issue with the lock though. Jeff, please have a look at the SDF, as well as this new guardian.
*CALEY. 5th attachment. This didn't have a diff when I started, but I think it got a diff when I cycled through the CALINJ system, as noted above. Reverted to the lower value, but Jeff, please have a look to see why this came back to the wrong value.
* SEIPROC, 6th attachment. The HAM1 FF switch being off was probably me, looking at things with Elenna on Friday. It is normally on, I believe, but Elenna, please have a look and make sure all the settings are as you want them to be.
After having done these, I put the IFO into Observing.