And here is the filter element, in an absolutely ridiculous condition, as Travis described it.

Here is a rough squence of today's upgrade, all times local PDT

07:22 rebuild H1EDC.ini to convince outselves the first DAQ restart will be model changes only

07:43 h1cdsrfm powered down, this breaks the linkage between Dolphin locations

07:43 h1ecatmon0 upgrade (reboot)

07:45 Dolphin network manager started on h1vmboot5-5, causing the standard set of end station systems to crash (susey, seiey, iscey and susex). We PAUSE'd the remaining EX (iscex, seiex)

07:51 Reboot h1susaux[ex, ey, b123, h2, h34, h56] pem[mx, my] to upgrade them. susaux[h34, ey] got stuck and were power cycled via IPMI.

08:03 DAQ 0-leg restart for new INI files across the board.

08:18 DAQ 1-leg restart. At this point omc0, susaux's and pemmid have good DAQ data, everyone else has BAD DAQ data.

08:27 Power down EX machines, power up EY machines. SWWD IPCs working, Dolphin IPC checks out.

08:32 Power up EX machines, all at the same time because of the Dolphin switch issue. They do not start. After some head scratching we remembered that the startup sequence needs to activate a dolphin switch port, which cannot happen at EX because the switch is damaged. Work around is for all three EX front ends to switch an unused port on the EY switch. Once this was put into place the EX machines started without anyone having to drive to the end station.

08:55 Reboot h1psl0 to upgrade PSL models (no dolphin cards, more about this later...)

08:56 Power down all non-upgraded corner station machines (SUS+SEI+ASC+LSC+OAF) but not h1omc0 (more about this later..)

09:00 h1psldbb is not running. It has an obsolete Dolphin IPC sender part in the model but no Dolphin card. RCG5.5.0 does not allow this. Rebuild model sans IPC part, starts running. Note PSLDBB DAQ data is BAD from this point till the second DAQ restart.

09:10 First power up h1sush2b, h1seih16 for HAM1 SEI work. SWWD IPC between the two working well (well, for these two, more later...)

09:20 Power up all remaining corner station computers

09:30 Discover wierd SWWD IPC receive values for HAM3 and HAM4 (val should be 1.0, but is -0.002 or 0.000).

09:34 try restart h1iopsush34, IPC values still bad. But h1omc0 has not been restarted so its using the old IPC configuration and could be writing out-of-bounds

09:35 restart h1omc0 models, SWWD IPC errors are resolved

09:44 power up h1cdsrfm. First EX is PAUSE'd, EY and CS are fenced. long range dolphin starts with no issues. A new MEDM is generated from the new H1.ipc file.

09:51 Complete in 2 hours.

Post install, note h1omc0 was already upgraded to 5.5.0 previously

The reason why the alignment was totally off is unknown. It was still off after turning on the PZT driver with an offset (2.5V) so it cannot be the PZT mirror. Something might have been bumped in the past month or two.

This was a sucess- this run through took only 7 minutes. I am shortening the 2 minute wait before increasing the gain to 90 seconds. If that still works, maybe we can go to 60 seconds.

To be more specific, the first attempt as described above meant the state took 6 minutes, 50 seconds. I loaded the change to reduce the wait time from 120 to 90 seconds, which only shortened the state length to 6 minutes, 30 seconds. The gain was only ramped to 8 for a very short period of time. I still think we can make this shorter, which we can do by making that wait time 60 seconds, and maybe taking bigger steps in the gain each time. However, we are still in the RCG upgrade, so I will hold off on changes to the guardian for now.

YAW3 is still limiting the length of the state. In this morning's relock, YAW3 convergence took nearly an additional minute more than the other loops. Once we have caught YAW3 up to everything else, we could make the state even shorter by raising the gain of other ADS loops. Two minutes of the state are taken up in the ramp of the A2L gain, so it is taking an additional 4 minutes, 30 seconds to wait for loop convergence.

Now it seems that PIT3 is taking too much time to converge, so I updated the guardian to also increase the PIT3 gain in the same way.

LVEA5 being off is expected, it's a pumped dust monitor so we turned it off for observing.

Attached is a screenshot of the grafana page, highlighting the 33 Hz calibration line, which seems to be the most sensitive to thermalization. Before, when the SRCL offset was set static, it appears that the 33 Hz line uncertainty starts at about 1.09 and then decays down to about 1.02 over the first hour. With the thermalization adjustment of the SRCL offset from 0 to -455 over one hour, the 33 Hz uncertainty starts around 0.945 and then increases to 1.02 over the first hour. Seems like we overshot in the other direction, so we could start closer to -200 perhaps and move to -455.

We decided to change the guardian so that it starts at -200 before then stepping its way up to -455 over the course of 75 minutes instead of 90 minutes.

With the update to the guardian to start at -200, each calibration line uncertainty has actually stayed very flat for these first 30 minutes of lock (except for the usual very large jump in the uncertainty for the first few minutes of the lock).

This shows the entire lock using the thermalization guardian with the SRCL offset ramping from -200 to -455, The line uncertainty holds steady the entire time within 2-3%!

Adding plot comparing the PSDs before and after getting rid of the peaks that can be indentified by this method.

These large BS PIT changes began 5th to 6th July 2024 (plot). This is the day shift from the time that the first lock like this happened 5th July 2024 19:26UTC (12:26PT): 78877 at the time we were doing PR2 spot moves. There also was a SUS computer restart 78892 but that appeared to be a day after this started happening.

Sheila, Camilla

This reminded Sheila of when we were heating a SUS in the past and causing the bottom mass to pitch and the ASC to move the top mass to counteract this. Then after lockloss, the bottom mass would slowly go back to it's nominal position.

We do see this on the BS since the PR2 move, see attached (top 2 left plots). See in the green bottom mass oplev trace, when the ASC is turned off on lockloss, the BS moves quickly and then slowly moves again over the next ~30 minutes, do not see simular things on PR3. Attached is the same plot before the PR2 move. And below is a list of other PR2 positions we tried, all the other positions have also made this BS drift. The total PR2 move since the good place is ~3500urad in Yaw.

• Different time May 21st to 24th 2024:
  • BS Oplev Drift
  • Plot shows 30urad M1 drift
  • PR2 Alignment Sliders P: 1435, Y: 1130
• Pre July 5th 2024:
  • No BS Oplev Drift
  • Plot shows 5urad M1 drift
  • PR2 Alignment Sliders P: 1565, Y: 3210
• July 5th 2024 to 6th Feb 2025:
  • BS Oplev Drift
  • Plot shows 50urad M1 drift
  • PR2 Alignment Sliders P: 1535, Y: 2785
• 6th Feb 2025 to 10th Feb 2025:
  • BS Oplev Drift
  • Plot shows 30urad M1 drift
  • PR2 Alignment Sliders P: 1480, Y: 1195
• 10th Feb 2025 to now:
  • BS Oplev Drift
  • Plot shows 30-40urad M1 drift
  • PR2 Alignment Sliders P: 1430, Y: -245

To avoid this heating and BS drift, we should move back towards a PR2 YAW of closer to 3200. But, we moved PR2 to avoid the spot clipping on the scrapper baffle, e.g.  77631, 80319, 82722, 82641.

I did a bit of alog archaeology to re-remember what we'd done in the past.

• In August of 2015, we found that we were struggling with PR3 pitch alignment jumping, then cooling down upon lockloss.  Alog 20268 talks about the implementation of the lock loss compensation, which first appeared in ISC_DRMI guardian in rev 11228.
• At some point (I didn't dig to find out when precisely), we also implemented the same filters for BS pitch.
• By Jan 2020, both BS and PRM had the soft ASC turn-off.
• In Jan 2020, ISC_DRMI rev 20905 we removed this soft ASC turn-off for both PR3 and BS.  The referenced alog 54709 notes that we shouldn't need those anymore, since we had installed wire heating baffles, to prevent the wires from being illuminated and heating up.
• We haven't had the soft turn-off filters in use since 2020, about 3 months before the end of O3b.  This may be why Camilla saw that we were seeing BS drift at the end of O3b.
• Perhaps our alignment during O4, until we moved the PR spots in May 2024, was such that we weren't susceptible to this wire heating.
• I don't think PR3 is seeing the same kind of trouble that it did back in 2015 upon lockloss, so I think its wire heating baffles are working as designed, so no need to make any changes to the PR3 controls.
• Sheila made the point that because we unclipped some of the +Y side of the beam (without moving the spot on the BS), maybe there is a bit more light that is illuminating the barrel of the BS or getting to the wires.  Or, something?  Without having looked at the actual drawings, I could imagine that the wire heating baffles are working better on PR3 than they are on the BS, because we hit PR3 much closer to normal incidence, whereas with the BS the light could be sneaking around the baffles.  Robert thinks that light could get inside the cage baffle and reflect around and be hitting and heating the wires.
• All of this seems to say that we should re-implement the soft ASC turn-off for the BS. I had a quick look at the 1/e time for the BS to move after lockloss (it's about 241 seconds), and the 1/e time for the filters (about 240 seconds, despite my quoting in alog 54706 that they were 25 min filters (2*pis are hard!)

To put back the soft turn-off of the BS ASC, I think we need to:

• Disable the BS M1 ASC lockloss trigger.  Jeff reminded me that this would foil my plans, since it turns off the ASC signals to the EUL2OSEM matrix.  This will mean that neither the Pit nor the Yaw BS M1 signals will be shut off by the lockloss trigger.  To disable, we'll need to set H1:SUS-BS_M1_TRIG_ASC_ENABLE to zero (which means that the ASC signals will always be passed to the EUL2OSEM matrix).  I don't think this is in guardian anywhere, so we should only need to change it and then accept in safe and observe snap files.
• Change ISC_DRMI around line 66 such that BS pit gain is not set to zero.  Also, have it turn off FM1 in addition to turning off the input.
• Change ISC_DRMI around line 141 to not hit the BS pit RSET button.

Camilla made the good point that we probably don't want to implement this and then have the first trial of it be overnight.  Maybe I'll put it in sometime Monday (when we again have commissioning time), and if we lose lock we can check that it did all the right things.

I've now implemented this soft let-go of BS pit in the ISC_DRMI guardian, and loaded.  We'll be able to watch it throughout the day today, including while we're commissioning, so hopefully we'll be able to see it work properly at least once (eg, from a DRMI lockloss).

This 'slow let-go' mode for BS pitch certainly makes the behavior of the BS pit oplev qualitatively different. 

In the attached plots, the sharp spike up and decay down behavior around -8 hours is how it had been looking for a long time (as Camilla notes in previous logs in this thread).  Around -2 hours we lost lock from NomLowNoise, and while we do get a glitch upon lockloss, the BS doesn't seem to move quite as much, and is mostly flattened out after a shorter amount of time.  I also note that this time (-2 hours ago) we didn't need to do an initial alignment (which was done at the -8 hours ago time).  However, as Jeff pointed out, we held at DOWN for a while to reconcile SDFs, it's not quite a fair comparison. 

We'll see how things go, but there's at least a chance that this will help reduce the need for initial alignments.  If needed, we can try to tweak the time constant of the 'soft let-go' to further make the optical lever signal stay more overall flat.

The SUSBS SDF safe.snap file is saved with FM1 off, so that it won't get turned back on in SDF revert.  The PREP_PRMI_ASC and PREP_DRMI_ASC states both re-enable FM1 - I may need to go through and ensure it's on for MICH initial alignment.

RyanS, Jenne

We've looked at a couple of times that the BS has been let go of slowly, and it seems like the cooldown time is usually about 17 minutes until it's basically done and at where it wants to be for the next acquisition of DRMI.  Attached is one such example.

Alternatively, a day or so ago Tony had to do an initial alignment.  On that day, it seemed like the BS took much longer to get to its quiescent spot.  I'm not yet sure why the behavior is different sometimes.

Tony is working on taking a look at our average reacquisition time, which will help tell us whether we should make another change to further improve the time it takes to get the BS to where it wants to be for acquisition.