FRS34149

Camilla and I removed the broken Model#VS35S2T0 Serial#3179 shutter and replaced it with Model#VS14S2T0 Serial#8796. The replacement unit had a slightly smaller outer diameter so we traded some posts with beam dumps that were on the side of the table not in use to get the correct height.

 Y-End TX Module Maintenance was performed today.
Everything went fairly well while following the instructions laid out in T1600436 -V12 Except the AOM Rejected Power measurement. This measurement inbetween PBSC2 and beam dump 2  is measuring the AOM power that we were dumping on to the beam dump. It didn't ever give us a static value like all the rest of the measurements. The max was a 18.5 mW  jump right after the we opened the shutter. Then the power would decay down to a lower value of 13.6 mw but over several minutes. The decay rate seemed to get longer after a while. The excitations were indeed turned off. We had zeroed the low power sensor with a shuttered background measurement. I'm not sure why that seemed to not be a steady number. We ended up taking the number that seemed like it had initially asymptote to before we saw the slow fall off.

The Beam Spots all look great. TX Sphere might be very slightly  to the right of perfectly center, but I think it's fine.
 

OLTF went well, Setup was well documented.

TITLE: 05/21 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: Busy day today! Arm gate valves are open, all HAM HEPI's are unlocked, and HAM1 continues to pump down. More activities below:

• After h1susb123 crashed (alog84500), all 18-bit DACs were upgraded to 20-bit: alog84508
  • 4x gain added to COILOUTFs on upgraded DACs: alog84509
  • MEDM screens updated: alog84514
• PSL PMC/FSS alignment: alog84512
• ALS-X beam profiling: alog84513
• h1sush7 crash and recovery: alog84521
• HAM1 cabling: alog84525

LOG:

The HAM1 floor cable tray near the PSL was reinstalled. Removed for the ISI installation. All cables on flanges D4 and D6 are dressed in front of the SEI cross beam. Cables are usually pulled behind the cross beam to avoid cables resting on the beam and to provide strain relief. More work needed to finish cable dressing/strain relief.

The following whitening controls concentrators D2400263 were installed:

ISC-R4, slot U30 S2400747
ISC-R4, slot U27 S2400748

ISC-R6, slot U33 S2500400
ISC-R6, slot U15 S2500401

F. Clara, J. Figueroa

J. Kissel

During today's upgrade of susb123's DACs (LHO:84519), and as Ryan and I were adding calibration change compensation in various OUTF banks (LHO:84509) I realized that we've never accounted for the gain loss when upgrading the PI DACs from 18-bit DACs to 20-bit DACs.

That "gain loss" more explicitly:
    - Assume you tune the open loop gain of the PI damping loop to be some magnitude |G|, with a 18-bit DAC, which has gain calibration of 20V/2^18 = 7.6294e-5 [V/ct].
    - Then you upgrade the DAC to a 20-bit DAC, which has gain calibration 20/2^20 = 1.9073e-5 [V/ct], a factor of 0.25x, or a factor of 4.0x less [V/ct].
    - This drops the loop gain |G| by the same factor of 4x, if not accounted for digitally.
Note, this is a scale factor at all frequencies. So, even if the DAC output is high-passed at 10 kHz, the gain above 10 kHz is still lower. 
I say this because they are high-passed at 10 kHz in both the ETM and ITM low-noise ESD drivers; see D1400301 block diagrams and circuit drawings D1500016 and D1600122 for ETM and ITM.

In every other SUS DAC chain, we compensate for this with a "20bitDAC" filter which is a "gain(4)" filter so that any upstream loop that uses that DAC chain doesn't have to be re-designed upon the transition from 18- to 20-bit DAC.

Here's the history of when each PI DAC was upgrade, including all the different names / numbers of how they're referred:
    IO Chassis    Card Name    "The nth DAC"   Slot    When Upgraded       aLOG 
    h1susex         DAC4           5th          7        Oct 2018          LHO:44918
    h1susey         DAC4           5th          7        Jun 2020          LHO:56217
    h1susb123       DAC6           7th          9        May 2025          LHO:84508

However, the PI models and MEDM interfaces aren't built with convenient places to apply a DAC gain calibration fix. An inconvenient but logical location to account for this would in each and every MODE's UPCONV_UC[n]_SIG bank, and I see no evidence of such a gain filter. If you wanted to be sneaky, you increase the OUT_MTRX coefficient from +/-1.0 to +/-4.0, but that has also not been done. I don't see any other evidence an additional gain of 4x along any mode's digital chain. 

I do note that we developed "EXTREME_PI_DAMPING" Dec 2024 LHO:81909 in hopes to get more oomph out of the ETMY PI system.
I wonder if we can get more oomph by just increasing the loop gain by a factor of 4x ...

I'm using this aLOG to mildly advocate for an ECR to implement some ESDOUTF filter banks just upstream of the DAC outputs, like we do for the audio-band DAC requests on the rest of the suspensions.

13:25:11 Wed 21 May 2025 PDT all models on h1sush7 stopped running. The SWWD to h1iopseih7 started its countdown with 100% IPC receive errors.

I set a bypass time of 999999 on h1iopseih7 to interrupt the countdown.

We first restarted all the models. This did not clear the error, and we found a PCI bus issue whereby only one of the four ADCs could be found.

As a precautionary measure we fenced h1sush7 from the Dolphin fabric.

Next we power cycled the computer. This fixed the PCI bus issues, all cards are visible and all models starting running.

I cleared the SWWDs and handed the system over to the control room.

Note, there were TIMING error flashes during this recovery time which we have not traced down.

[Wed May 21 13:25:11 2025] h1iopsush7: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Wed May 21 13:25:11 2025] h1susauxh7: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Wed May 21 13:25:11 2025] h1sussqzin: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Wed May 21 13:25:11 2025] h1susfc1: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
 

J. Kissel
E1900216 (IIET:13232)
E2100485 (IIET:20828)

In light of the last 24 hours of 20-bit DAC upgrades (see LHO:84519), it's worth updating my Oct 2023 (LHO:73518) list of priorities for DAC upgrades:

    (a) [2x] Upgrade PRM M2/M3, and PR2 M2/M3. 
        DAC5 (Slot 8, card_num = 5) on h1sush2a for PRM
        DAC4 (Slot 7, card_num = 4) on h1sush34 for PR2
    (b) [2x] Upgrade PR3/SR3 M2/M3 DACs
        DAC6 (Slot 9, card_num = 6) on h1sush2a for PR3
        DAC3 (Slot 6, card_num = 3) on h1sush56 for SR3
    (c) [1x] Upgrade ITMX and ITMY UIM DACs
        DAC4 (Slot 8, card_num = 4) on h1susb123
    (d) [3x] Finish out susex IO chassis upgrades (which is ETMX M0 / R0 and TMSX M1)
        DAC0, DAC1, DAC2 on h1susex
    (e) [3x] Finish out susey IO chassis upgrades (which is ETMY M0 / R0 and TMSY M1)
        DAC0, DAC1, DAC2 on h1susey
    (f) [5x] Finish out susb123 IO chassis upgrade (which is ITMX M0/R0, ITMY M0/R0, and BS M1)
        DAC0, DAC1, DAC4, DAC5, DAC9 on h1susb123
    (g) [3x] MC2 M2/M3, MC1 M2/M3, MC3 M2/M3
        DAC3 (Slot 7, card_num = 3) on h1sush34 for MC2
        DAC3 (Slot 6, card_num = 3) on h1sush2a for MC1
        DAC4 (Slot 7, card_num = 4) on h1sush2a for MC3
    (i) [3x] MC1/MC3/PRM/PR3 top masses
        DAC0 (Slot 2, card_num = 1) in h1sush2a for MC1/MC3 M1
        DAC1 (Slot 4, card_num = 1) in h1sush2a for MC3/PRM M1
        DAC2 (Slot 5, card_num = 2) in h1sush2a for PRM/PR3 M1
    (j) [3x] MC2/PR2/SR2 top masses
        DAC0 (Slot 2, card_num = 0) in h1sush34 for MC2/PR2 M1 
        DAC1 (Slot 4, card_num = 1) in h1sush34 for PR2/SR2 M1
        DAC2 (Slot 5, card_num = 2) in h1sush34 for SR2 M1
    (k) [4x] SRM/SR3/OMC top masses
        DAC0, DAC1, DAC2, DAC3 in h1sush56
    (l) [2x] IMs
        DAC0, DAC1 in h1sush2b

25 cards left to go!

J. Kissel, F. Clara, E. von Ries, D. Barker, E. Dohman, O. Patane, R. Short

After fixing MEDM screens, I was able to validate that all expected drive signal channel assignments reach the expected DAC outputs for H1SUSITMY, H1SUSITMX, and H1SUSBS. 

As such, I restored the function of top mass damping loops, L2 to R0 tracking for the ITMs, and optical lever damping for the BS.

We've confirmed that alignment offsets have been restored.

With the SUS happily damping, I then restored the seismic isolation system, using the SEI guardians to bring HEPIs and ISIs back to FULLY_ISOLATED for the ITMs and FULLY_ISOLATED_NO_ST2_BOO for the BS.

Executive Summary of the last 24 hours of events regarding h1susb123 DAC card failure:
    - LHO:84500 h1susb123's Slot 4, DAC 2 -- an 18-bit DAC -- died and caused the front-end to crash, which eventually tripped all BSC1, BSC2, and BSC3 seismic systems via Software Watchdog (SWWD) 
    - LHO:84506 Having agreed that we should take the opportunity to enact ECR E1900216 (IIET:13232) and upgrade all the DAC cards in the chassis to 20-bit DACs -- we prepped the h1susb123 user and IOP models for the change 
    - LHO:84508 Erik and Fil replace all 18-bit DAC cards with 20-bit DAC cards, Dave compiles, installs, and restarts the h1susb123 front-end's models with the new code.
    - LHO:84509 With the models running, Ryan and I installed updates to the COILOUTF filters to account for the calibration change between 18 and 20 bit DACs.
    - LHO:84514 I cleaned up the MEDM interface for the SUS-ITMs and SUS-BS to ensure they accurately show signal flow from user model output to the DAC.
    - (this aLOG) Having validated signal flow, we restored the SEI and SUS systems to full nominal functionality.

This post summarizes some features of the 106.06736 comb’s history over O4 so far. This comb was found in the run-averaged spectrum (Fig.1) while updating the lines lists. Its cause is unknown. 

• There are several peaks over the course of the run in the 106 Hz region. Not all of these peaks occur at the same time.
• For some of these peaks, it appears that the peaks entirely disappear during breaks occurring at least one month later. For others, the peaks continue after the breaks, as evidenced by the attached weekly spectrogram (Fig.2).

Attached are static images of plots displaying the comb’s structure and behavior. Fig.1 is the run-averaged spectrum of the comb and the other two (Figs.2 and 3) are spectrograms of it. The blue spectrogram in Fig.2 is weekly over the ongoing O4 run and the green spectrogram in Fig.3 is daily.

Temporary HAM1 gauge EPICS channel

In lieu of the Beckhoff IOC, the HAM1 gauge raw voltage signal was converted to a pressure channel by a new EPICS IOC. This channel was added to the DAQ for trending, EDC+DAQ restart was needed

DAQ Restart

For above HAM1 vac channel. Also H1EPICS_PWRSTRIP.ini chans were added.

J. Kissel

Even though we confirmed that no DAC's "EPICs-channel-to-physical-card-assignment" changed for the model updates that correspond to the 20-bit DAC upgrade of h1susb123 IO chassis this morning (LHO:84506), the MEDM overview screens showed that channels were askew as we started to turn control of the suspensions back on. 

I realize that this is a result how the RCG automatically generates channel numbers for DACs in the USER MODEL.

Let's use ITMX as the demonstration, because that ended up being the most jumbled model.

Physical Slot      IOP Model Name    IOP Card Num     1st IOP Channel    USER Model Name   USER Card Num    GDSTP MEDM Label   1st USER Channel
2                  DAC_0             0                0_0            
4                  DAC_1             1                1_0                DAC_1             1                D1                 0_0
5                  DAC_2             2                2_0            
6                  DAC_3             3                3_0
7                  DAC_4             4                4_0                DAC_4             4                D4                 1_0
8                  DAC_5             5                5_0                DAC_5             5                D5                 2_0
9                  DAC_6             6                6_0     
10                 DAC_7             7                7_0                DAC_7             7                D7                 3_0
So -- even though the IOP model, USER model, and even the automatically generated screens agree to use the call the name of these DAC cards 1, 4, 5, and 7, the actual underlying channels that display the output of these DACs call them DAC 0, 1, 2, and 3.

In a mixed chassis, this USER model DAC channel number counting starts and continues as a simple number of cards called out in the model, first counting 18s, then 20s, then 16s (I think).
So, since the number of 18s in this formerly mixed chassis dropped to zero, that channel count got re-arranged.

This meant that the display on the SUS_OVERVIEW screens (e.g. this screenshot of the ITMX M0 and L1 stages) would not correctly show a given OSEMs signal smoothly transitioning from the "MASTER OUTPUT" column to "USER MODEL DAC OUTPUT" to "IOP MODEL DAC OUTPUT."

These channel assignments are defined in the MEDM macro text files; for this chassis these are 
    /opt/rtcds/userapps/release/sus/common/medm
        susitmy_overview_macro.txt
        susitmx_overview_macro.txt
        susbs_overview_macro.txt
These are nominally common files between sites. Regrettably, because LLO upgraded their 18s to 20s at a different rate than LHO, they've necessarily diverged, so they're not committed / tracked in svn. So at the start of today it was locally modified (due to its difference with LLO, which uses the committed version), and now I've made the updates and it's locally modified (and they're still different from the committed version), so no good version control diffs.

I've updated the macro and validated with simple offsets that each and every output signal goes through the three different MASTEROUT USERDAC and IOP DAC channels for ITMY, ITMX and BS.

In hopes to have *something* in the future, I attach the updated macro files here.

Sheila, Raed, Camilla. WP 12559. Repeating work at EY: 81358

The EX ALS shutter wasn’t all way closed when we arrived, even after opening and closing it’s the stayed the same, partially closed, see photo in "closed" position and video. Opened FRS issue #34149 .

Attached the beamscan data we took. Beam looked nice and gaussian even though it was astigmatic. Example photo attached, I have all other photos.

We again checked the HWS beam that we don't like much and confirmed that the shape wasn't Gaussian, it looked like a top hap and the HWS beam profile is attached soon after the collimator. 

Before commissioners need the main PSL beam for alignment work, I decided to touch up the PMC and RefCav alignments this morning as they came back worse off after having the PMC unlocked over the weekend.

With the ISS off, I started with PMC alignment:

• PMC start: 104.9 W
• PMC end: 105.7 W

After engaging the ISS and ensuring it was diffracting around 4% (no RefSignal adjustment needed), I moved on to RefCav alignment:

• TPD start: 0.770 V
• TPD end: 0.820 V

Decent improvements, but judging by the RefCav REFL spot, it's possible an on-table alignment touchup could help further (largely in pitch). I'll also want to recalibrate the rotation stage at some point before IFO locking starts, but that'll likely have to wait until we can send high power through to the MC REFL dumps and not onto the light pipe shutter.

Wed May 21 10:06:29 2025 INFO: Fill completed in 6min 26secs

J. Kissel, R. Short

Given that h1susb123 had an emergency upgrade to 20-bit DACs this morning, we need to do "the usual" cheat to ensure the upstream loops that use the stages that got upgrade don't notice a difference -- add a "20bitDAC" gain of 4.0x to the COILOUTF banks in FM10 of any stage that got the upgrade from 18- to 20-bit DACs. This accounts for the change in calibration from  20V / 2^18 to 20V / 2^20.

This has been done for the
    - H1 SUS ITMY M0 F1F2F3SD
    - H1 SUS ITMY R0 F1F2F3SD
    - H1 SUS ITMY M0 LFRT / R0 LFRT
    - H1 SUS ITMY L1 ULLLURLR

    - H1 SUS ITMX M0 F1F2F3SD
    - H1 SUS ITMX R0 F1F2F3SD
    - H1 SUS ITMX M0 LFRT / R0 LFRT
    - H1 SUS ITMX L1 ULLLURLR

    - H1 SUS BS M1 F1F2F3LF
    - H1 SUS BS M1 RTSDxxx
COILOUTF banks, corresponding to all the stages that got upgraded this morning.

Attached are the automatically generated "diff" screens for each model showing that I added the gain(4) design string in each 9th module (counting starting a 0, that's FM10) to all of the above mentioned OSEMs.
I then hit "LOAD_COEFFICIENTS" on each user model's GDS_TP screen (which clears the diff).

The filter file changes have been committed to
    /opt/rtcds/userapps/release/sus/h1/filterfiles
        H1SUSBS.txt
        H1SUSITMX.txt
        H1SUSITMY.txt
as of rev 31491.

Ryan accepted FM10 being ON for all of the above mentioned OSEM chains in the 
    h1susitmy_down.snap
    h1susbs_down.snap
    h1susitmx_down.snap

    and
   
    h1susitmy_observe.snap
    h1susitmx_observe.snap
    h1susbs_observe.snap
as of rev 31492.

WP12562

Jonathan, Erik, EJ, Jeff, Ryan, Fil, Dave:

As part of the h1susb123 crash investigation, after power cycling the computer and IO Chassis the next thing to do was to replace the 2nd 18bit-DAC card which was no longer visible on the bus.

During the 08:30 vent meeting it was discussed which option we should pursue: replace the suspect 18bit-DAC with another 18bit-DAC, replace this one card with a 20bit-DAC, replace all six 18bit-DACs with 20bit-DACs. The third option was chosen.

Hardware:

Keep the exisiting two 20bit-DACs, replace the six 18bit-DACs with 20bit-DACs. Put aside the second 18bit-DAC as a suspect card, put the other five cards into the O4 spare pool.

Sofware:

h1iopsusb123:

replace the DAC area with eight 20bit-DACs from CDS_PARTS, card numbers 0-7. The SWWD system already DACKILLs all eight DACs (card_num 0-7) irregardless of which suspension trips, so no change needed.

No change to INI file.

h1susitmx, h1susetmx, h1susbs, h1susitmpi:

See Jeff's alog for model changes. DAC part naming was chosen such that there were no changes to INI files.