WP12061 Upgrade RCG h1susex, remove LIGO-DAC delays

EJ, Erik, Jonathan, Dave, Daniel, Marc:

h1susex RCG was upgraded to a custom 5.3.0 specifically for the IOP to remove a delay in the new LIGO 28AO32 DAC. We compliled all of the user models with this RCG as well.

Our first restart was a simple model restart, but we got Dolphin errors. So our second restart was to fence h1susex from the Dolphin fabric and power cycle h1susex via IPMI. After this the models started with no issues.

No DAQ restart was required.

Code path is /opt/rtcds/rtscore/advLigoRTS-5.3.0_dacfix

WP12063 Alert System

Dave:

The locklossalert system code was modified to permit an alert window which spans over midnight, needed for the new owl shift hours.

Note that the business/every day filter is applied after the minute-in-day filter, so a window starting Friday evening and extending into Saturday morning will cut off at midnight if business days only is selected.

One other change:

For everyone subscribed to Guardian alerts, if for any reason the Guardian EPICS record cannot be read (node down, h1guardian1 down) the alert will now default to SEND.

Guardian Reboot

TJ:

h1guardian1 was rebooted at 08:21 PDT. All nodes except TEST came back automatically. TJ worked on TEST and got it going.

MSR Rack Cleanup

Jonathan:

Jonathan removed two test switches from the MSR racks which are no longer needed.

I removed two switches from the h1-daq-1 rack that had been put in for testing.  They are no longer needed in racks.

I also took a moment to sort and clean up a number of SFP transceivers that had been left out on the MSR work area.

Tue Aug 27 08:08:21 2024 INFO: Fill completed in 8min 17secs

Jordan confirmed a good fill curbside.

TITLE: 08/27 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 3mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY: Locked for 2.5 hours, seems like an automated relocked before that. PEM magnetic injections are running now. Planned 4 hours of maintenance today.

Workstations were updated and rebooted.  This was an OS packages update.  Conda packages were not updated.

TITLE: 08/27 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: Observing and have been Locked for 5.5 hours. Quiet evening with nothing to report. Feedforward measurements were run, and I also just tuned squeezing up.

Little comparison on the damping for new PI over the past 5 days (ndscope)
Since starting back up last week, we have been seeing a new PI (79665), and it had caused a few locklosses. On Friday afternoon (08/24 ~00:00UTC - crosshairs on ndscope) it was decided that the easiest way to try and control it for now would be to change our input power from 60W to 61W. This, along with some changes to the damping gain, has helped keep it under control. Comparing the PI's behavior to the length of our lock stretches, we only see the ringups during the first 1.5-2 hours of each lock, so once we're thermalized past a certain point, they stop happening. The changes to damping (I'm not sure what was done besides raising the damping gain) also means that they get caught and damped quickly.
 

LOG:

23:30 Observing and Locked for 2 minutes
04:27 Dropped Observing to take FF measurements
04:46 Feedforward measurements done, running SQZ alignment
04:55 Back to Observing

TITLE: 08/26 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 6mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY: Observing and just got to NLN a few minutes ago. Once we're thermalized I'll be taking LSC FF measurements.

TITLE: 08/26 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 139Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: The shift was mostly taken up by commissioning time. There was recovery from a CDS crash this morning, but recovery was pretty straight forward. The reacqusitions from the two lock losses one needed an initial alignment, the other did not. Minor interventions for both.
LOG:

• 1744 Back to low noise, time for commissoining
• 1827 Lock loss
  • Tried relocking without an initial alignment, didn't work. Fully auto IA and main locking
• 2059 Observing
• 2159 Lock loss
  • No initial alignment, but I held at Locking_Arms_Green because it would move to the next state before the WFS would converge and cause a lock loss. We then lost lock at Engage_ASC_For_Full_IFO, and the next time it made it up on its own.                             

I restarted the 4 SUS in lock charge nodes and the PEM magnetic injection node to reestablish the connection to the front ends that can go away after front end crashes, like the one this morning ( alog79708). We wanted to be prepared for these to run tomorrow morning if the IFO is locked.

FAMIS 27796

• TCSx chiller: Level of 29.8; did not add water
• TCSy chiller: Level of 9.8; did not add water
• No water seen in small cup under the "slow leak"
• Values updated in T2200289

Before the vent, we had lowered the DARM offset used at the end of the DARM_OFFSET state for locking the OMC since we had seen the PRG fall off and cause a lockloss with the nominal offset of 9e-05 (see alog79082 for details). When locking this afternoon, I raised the offset from 6e-05 back to 9e-05 after running through DARM_OFFSET, and seeing that the PRG didn't plummet and cause a lockloss, we continued locking. The OMC locked on the first try, something that hasn't been the case recently, so having more carrier light there seems to help. I compared OMC scans from this lock against the last lock, which used the lower DARM offset; attachment 1 shows the scan with the higher offset and attachment 2 with the lower offset. According to the OMC-DCPD_SUM channel, we get ~10mW more carrier light on the OMC when locking with this higher DARM offset.

I've put this DARM offset of 9e-05 back into ISC_LOCK and loaded it. We can watch over the next couple of lock acquisitions to see if the problem wth the PRG dropping off resurfaces.

I measured OMC fringe wrapping after OFI replacement. First I ran the dtt template in 78942, which caused a lockloss. In the next try, I reduced the excitation from 3421 to 600, which corresponds to 2.4 um pp OMC L (H1:SUS-OMC_M1_DAMP_L_IN1_DQ). The attachment shows that OMC scatter is worse than one in April (blue: April vs purple: today). We may need to adjust the OFI temperature.

FAMIS 26290


Laser Status:
    NPRO output power is 1.831W (nominal ~2W)
    AMP1 output power is 64.55W (nominal ~70W)
    AMP2 output power is 137.1W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 6 days, 23 hr 49 minutes
    Reflected power = 21.13W
    Transmitted power = 105.6W
    PowerSum = 126.7W

FSS:
    It has been locked for 0 days 1 hr and 37 min
    TPD[V] = 0.6479V

ISS:
    The diffracted power is around 1.9%
    Last saturation event was 0 days 1 hours and 37 minutes ago

Possible Issues:
    AMP1 power is low
    PMC reflected power is high
    FSS TPD is low
    ISS diffracted power is low

FAMIS 21189

Nothing much of note this week. PMC REFL is still increasing slowly while PMC TRANS is decreasing.

The FSS TPD signal is still low, and since I wasn't able to increase it much last week, we plan to go into the enclosure and tune up the FSS path on-table soon to fix it.

There were a few PCAL SDF diffs when we made it back to low noise today at CALEY. It looked like these were loaded into epics from teh safe.snap and these numbers disagreed with what was saved in the observe.snap file. I confirmed with Franscisco that this was the case, and he had me verify that they agreed with alog77386. I then also saved these new values in the safe.snap file as well. Interestingly, these channels are not monitored in the safe.snap.

TJ, Jonathan, EJ, Dave:

Around 01:30 this morning we had a Dolphin crash of all the frontends at EY (h1susey, h1seiey, h1iscey). h1susauxey is not on the Dolphin network as was not impacted.

We could not ping these machines, but were able to get some diagnostics from their IPMI management ports.

At 07:57 we powered down h1[sus,sei,isc]ey for about a minute and then powered them back on.

We checked the IX Dolphin switch at EY was responsive on the network.

All the systems came back with no issues. SWWD and model WDs were cleared. TJ is recovering H1.

We can edit this list as needed. 

We are having trouble locking this afternoon because of the wind, but we have some requests for the operators this weekend if they are able to relock. 

We've changed the nominal locking power to 61W, in the hopes that this might avoid the PI or let us pass through it quickly.

When we first get to NLN, please take the IFO to NLN_CAL_MEAS and run a calibration sweep.  If we stayed locked for ~1.5 hours, please re-run, and if we are ever locked for more than 3 hours please re-run again. 

After the calibration has run, we would like to check the camera set points since we have seen earlier today that they are not optimal and that might be related to our PI problem.  We already updated the offset for PIT1, but we'd like to check the others.   We modified the script from 76695 to engage +20 dB filters in all these servos to speed up the process.   Each DOF should take a little more than 15 minutes.  We'd like these run with the POP beam diverter open, so we can see the impact on POP18 and POP90.  The operators can run this by going to /ligo/gitcommon/labutils/beam_spot_raster and typing python camera_servo_offset_stepper.py 1 -s now  (and once 1 completes, run 2 and 3 if there's time.)

I've added 3 ndscope templates that you can use to watch what these scripts do.  the templates are in userapps/asc/h1/templates/ndscope/move_camera_offsets_{BS, DOF2_ETMX, DOF3_ETMY}.yml  We'd like to see if any of these can increase the arm powers, or POP18.  If there is a better value found, it can be set to the default by updating lscparams.py lines 457 to 465.