Before commissioners need the main PSL beam for alignment work, I decided to touch up the PMC and RefCav alignments this morning as they came back worse off after having the PMC unlocked over the weekend.
With the ISS off, I started with PMC alignment:
After engaging the ISS and ensuring it was diffracting around 4% (no RefSignal adjustment needed), I moved on to RefCav alignment:
Decent improvements, but judging by the RefCav REFL spot, it's possible an on-table alignment touchup could help further (largely in pitch). I'll also want to recalibrate the rotation stage at some point before IFO locking starts, but that'll likely have to wait until we can send high power through to the MC REFL dumps and not onto the light pipe shutter.
Wed May 21 10:06:29 2025 INFO: Fill completed in 6min 26secs
J. Kissel, R. Short
Given that h1susb123 had an emergency upgrade to 20-bit DACs this morning, we need to do "the usual" cheat to ensure the upstream loops that use the stages that got upgrade don't notice a difference -- add a "20bitDAC" gain of 4.0x to the COILOUTF banks in FM10 of any stage that got the upgrade from 18- to 20-bit DACs. This accounts for the change in calibration from 20V / 2^18 to 20V / 2^20.
This has been done for the
- H1 SUS ITMY M0 F1F2F3SD
- H1 SUS ITMY R0 F1F2F3SD
- H1 SUS ITMY M0 LFRT / R0 LFRT
- H1 SUS ITMY L1 ULLLURLR
- H1 SUS ITMX M0 F1F2F3SD
- H1 SUS ITMX R0 F1F2F3SD
- H1 SUS ITMX M0 LFRT / R0 LFRT
- H1 SUS ITMX L1 ULLLURLR
- H1 SUS BS M1 F1F2F3LF
- H1 SUS BS M1 RTSDxxx
COILOUTF banks, corresponding to all the stages that got upgraded this morning.
Attached are the automatically generated "diff" screens for each model showing that I added the gain(4) design string in each 9th module (counting starting a 0, that's FM10) to all of the above mentioned OSEMs.
I then hit "LOAD_COEFFICIENTS" on each user model's GDS_TP screen (which clears the diff).
The filter file changes have been committed to
/opt/rtcds/userapps/release/sus/h1/filterfiles
H1SUSBS.txt
H1SUSITMX.txt
H1SUSITMY.txt
as of rev 31491.
Ryan accepted FM10 being ON for all of the above mentioned OSEM chains in the
h1susitmy_down.snap
h1susbs_down.snap
h1susitmx_down.snap
and
h1susitmy_observe.snap
h1susitmx_observe.snap
h1susbs_observe.snap
as of rev 31492.
WP12562
Jonathan, Erik, EJ, Jeff, Ryan, Fil, Dave:
As part of the h1susb123 crash investigation, after power cycling the computer and IO Chassis the next thing to do was to replace the 2nd 18bit-DAC card which was no longer visible on the bus.
During the 08:30 vent meeting it was discussed which option we should pursue: replace the suspect 18bit-DAC with another 18bit-DAC, replace this one card with a 20bit-DAC, replace all six 18bit-DACs with 20bit-DACs. The third option was chosen.
Hardware:
Keep the exisiting two 20bit-DACs, replace the six 18bit-DACs with 20bit-DACs. Put aside the second 18bit-DAC as a suspect card, put the other five cards into the O4 spare pool.
| Adnaco-slot | A3-4 | A3-3 | A3-2 | A3-1 | A2-4 | A2-3 | A2-2 | A2-1 | A1-4 | A1-3 | A1-2 | A1-1 | ||
| was | 20bit DAC-1 | 18bit 101208-60 | 18bit 101208-09 | 18bit 110425-20 | 18bit 110425-48 | 20bit DAC_0 | 18bit 110425-23 | ADC-1 | 18bit 110425-32 | ADC-0 | timing | |||
| is now | 20bit DAC | 20bit 220218-29 | 20bit 210303-49 | 20bit 210303-35 | 20bit 210303-22 | 20bit DAC | 20bit 210303-57 | ADC-1 | 20bit 220218-08 | ADC-0 | timing |
Sofware:
h1iopsusb123:
replace the DAC area with eight 20bit-DACs from CDS_PARTS, card numbers 0-7. The SWWD system already DACKILLs all eight DACs (card_num 0-7) irregardless of which suspension trips, so no change needed.
No change to INI file.
h1susitmx, h1susetmx, h1susbs, h1susitmpi:
See Jeff's alog for model changes. DAC part naming was chosen such that there were no changes to INI files.
Wed21May2025
LOC TIME HOSTNAME MODEL/REBOOT
07:29:37 h1susb123 ***REBOOT*** <<< try power cycle computer
07:32:01 h1susb123 h1iopsusb123
07:51:06 h1susb123 ***REBOOT*** <<< try power cycle computer and IO Chassis
07:53:30 h1susb123 h1iopsusb123
09:53:47 h1susb123 ***REBOOT*** <<< replace all 18bit-DACs with 20bit-DACs
09:56:18 h1susb123 h1iopsusb123
09:56:31 h1susb123 h1susitmy
09:56:44 h1susb123 h1susbs
09:56:57 h1susb123 h1susitmx
09:57:10 h1susb123 h1susitmpi
D. Barker, E. Dohmen, J. Kissel As a result of the failure of h1susb123's DAC_1, card_num = 1 (with counting starting at 0), in Slot 4 of the IO chassis which drives ITMX and ITMY's reaction chain R0 F1F2F3SD OSEMs (see LHO:84500 and call-out of the card in this screenshot of D1301004-v4) we've agreed to "just" upgrade the entire chassis to 20-bit DACs per E1900216 (IIET:13232). After a careful review of the user model layout of this very confusing mixed IO chassis -- with DAC cards shared across suspensions coupled with an barely intuitive order of which stage is assigned to which card -- Dave, EJ, and I have confirmed that the following AFTER screetshots show the correct DAC (re)assignments for upgrading the entire h1susb123 chassis to 20 bit DACs. - h1susitmy BEFORE vs AFTER - h1susitmx BEFORE vs AFTER - h1susbs BEFORE vs. AFTER, and - h1susitmpi BEFORE vs. AFTER. I also attach the BEFORE as-built card layout from D1301004-v4, and the "this is the truth" map of which stage uses what card from the redlines written in the abstract of D1100022-v13. These models are ready for install and have been committed to /opt/rtcds/userapps/release/sus/h1/models/ h1susbs.mdl h1susitmpi.mdl h1susitmx.mdl h1susitmy.mdl as of userapps svn repo version 31490.
TITLE: 05/21 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 9mph Gusts, 6mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.17 μm/s
QUICK SUMMARY: HAM1 continues to pump down, pressure currently around 1.3e-5 Torr. HAM3 cleanroom move has started in the LVEA. Working now with CDS team to diagnose and fix last night's h1susb123 crash.
At 21:13:58 we had a crash of h1susb123 (ITMX, ITMY, BS) which in turn caused a SWWD DACKILL of the seismic front ends for these chambers.
Unlike the h1susb123 crash Sunday 11 May 2025 in which lscpi was slow but showed good cards, in this case lspci is responsive and shows a possible problem with the 2nd 18bit-DAC card.
Like the Sunday crash, dmesg does not give much information (listing below).
First thing to try would be a power cycle of the IO Chassis to see if the 18bit DAC is OK. Given the lateness of the hour we could delay this until the morning.
[Tue May 20 21:13:56 2025] rts_cpu_isolator: LIGO code is done, calling regular shutdown code
[Tue May 20 21:13:56 2025] h1iopsusb123: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Tue May 20 21:13:57 2025] h1susitmpi: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Tue May 20 21:13:57 2025] h1susbs: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Tue May 20 21:13:57 2025] h1susitmy: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Tue May 20 21:13:57 2025] h1susitmx: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
To get the DAC drives running again on h1seib[1,2,3] I've bypassed the SWWDs for these front ends.
Power Cycle Computer ----------------------------------------------
EJ, Dave:
First thing to try was a power cycle of the front end computer.
07:26 Fence h1susb123 from Dolphin fabric, stop models, power down cpu
07:28 Power h1susb123 back up using IPMI
At this point lspci is reporting the 2nd 18bit-DAC slot as empty, as expected IOP model stops with insufficient cards.
So, looking like bad slot or bad DAC card.
Power Cycle Computer And IO Chassis -----------------------------------------------
EJ, Jonathan, Dave:
Second thing to try, power down computer as above, then power cycle the IO Chassis for about 30 seconds.
Result: the same, lspci does not show anything in the slot the 2nd 18bit-DAC is in.
HAM1 pressure has already gone below the threshold that we consider the suspensions to stop floating, so I've taken in vac transfer functions for RM1, RM2, and PM1. All three suspensions are looking healthy.
Earlier today, Fil completed the coil driver swap for PM1 (84490), so I updated the COILOUTF filters(84499), and now all filters (physical and digital) match the ones for the RM coil drivers (84499).
Note that these were taken with the ISI locked and HEPI offline.
RM1 (r12326(data), r12327(results)):
Data: /ligo/svncommon/SusSVN/sus/trunk/HTTS/H1/RM1/SAGM0/Data/2025-05-21_0000_H1SUSRM1_M1_WhiteNoise_{L,P,Y}_0p01to50Hz.xml
RM2 (r12329):
Data: /ligo/svncommon/SusSVN/sus/trunk/HTTS/H1/RM2/SAGM0/Data/2025-05-21_0015_H1SUSRM1_M1_WhiteNoise_{L,P,Y}_0p01to50Hz.xml
PM1 (r12328):
Data: /ligo/svncommon/SusSVN/sus/trunk/HTTS/H1/PM1/SAGM0/Data/2025-05-20_2200_H1SUSRM1_M1_WhiteNoise_{L,P,Y}_0p01to50Hz.xml
Results: /ligo/svncommon/SusSVN/sus/trunk/HTTS/H1/PM1/SAGM0/Results/2025-05-20_2200_H1SUSRM1_M1*
Comparison of all three HTTS (r12330):
Yesterday I fixed the PM1 COILOUTF filters because we had accidently copied in filters from the RMs, which had a different coil driver set up due to an ECR that hadn't yet been done for PM1, so I corrected them (84455). Today, Fil changed out the PM1 coil driver so it now is correct as per ECR E2200307 and E2400048 (84490), so I've re-re-placed the PM1 coil driver filters with the now correct same filters that the RMs have. I've loaded them in.
Jordan, Travis After yesterday's preparation works (see aLog 84475) the pumpdown procedure was started on HAM1. 2 purge chamber-air measurement was done, one was during purging; the purge air dew point was -40.3 deg C; the blowdown air dew point was -40.1 deg C. This is very dry, so a very good start for the pumpdown. The pumpdown was started with the Supersucker cart's 500 l/min Anest Iwata scroll pump, then at <500 mTorr, the recently installed 500 l/s Leybold turbo was switched on. As the pressure dropped down to the E-5 Torr region, the purging all metal valve on the top of the chamber (valve for the new gauge) was closed. In the meantime, the HAM1 gauge was wired (see aLog 84494). The channel was also added to DAQ for trending, and can be seen on the MEDM screen. Thanks for all of this to Fil, Dave, and Patrick! At the moment, the pressure in HAM1 is 6.5E-5 Torr, tomorrow the chamber must be ready for leak check. In preparation for leak checking, all the double O-ring flanges were wrapped with Kapton-tape, to avoid flooding the annulus volume with He during leak checking - here note that because the feedthrus are all populated with cables, bagging of flanges will be hardly possible, so the area will be flooded with He anyways.
Tony, Jim
This afternoon we replaced the H3 L4C on HAM2 that I found broken a couple weeks ago. First we disconnected the actuator from the HEPI foot and slid it back onto a set of stairs that just happened to be the right height. Then we removed the spacers on the foot and pulled the pier chassis out of the way to allow access into the foot to release the clamp block holding the L4C in place. The access to this clamp was difficult, and took some patience with a small ratchet to get loose enough to release the L4C. Put the new L4C in and put the pier back together. The new L4C is working, but we were looking at the wrong IPS when tightening the actuator back up. I will have to go out tomorrow morning and loosen the actuator and slid it back to recenter the IPS. Not too hard, HEPI is locked , which will preserve the mechanical location.
With the temporary HAM1 gauge PV created today and the recent HAM6 gauge upgrade VACSTAT is back to a full complement of gauges (first time since 26 Dec 2024).
Note that HAM1's slope and delta-p settings are currently non-nominal while it is pumping down, but it will alert if its vacuum glitches.
WP 12543
The new HAM1 vacuum gauge is powered up and connected to Beckhoff. Network cable to gauge is connected to the CU1128 Beckhoff hub in the biergarten (SUS-R6). One of the existing Pirani/cold cathode cables was used to provide the 24V from the LY vacuum rack.
Software configuration of the gauge is needed to read out the pressure from the Beckhoff system. Used PT100B cable to temporarily send analog signal from gauge to LY Rack to monitor pump down. See alog 84482 for details.
TITLE: 05/20 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: HAM1 is now pumping back down and will go overnight. Binary and coil driver chassis for RMs and PM1 were updated, see alog84490. The broken L4C at HAM2 is currently being replaced. All optics in HAMs 1 & 2 are now ALIGNED.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 14:39 | FAC | Kim, Nellie | LVEA | N | Technical cleaning | 16:06 |
| 15:37 | VAC | Jordan | LVEA | N | Purge air and pump checks | 15:53 |
| 15:50 | CDS | Fil | LVEA | N | VAC and ISC cabling | 19:05 |
| 16:49 | CDS | Jackie | LVEA | N | VAC and ISC cabling | 19:05 |
| 17:00 | VAC | Janos | LVEA | N | Checking pumps | 17:10 |
| 17:02 | FAC | Kim, Nellie | EX, EY | N | Restocking garb | 17:39 |
| 17:19 | AOS | Betsy | LVEA | N | Cleanup around HAM3 | 18:11 |
| 17:26 | SUS | Rahul | LVEA | N | Toolpan cleanup around HAM3 | 17:31 |
| 17:57 | ISC | Camilla | LVEA | N | Toolpan cleanup around HAM3 | 18:33 |
| 17:58 | FAC | Nellie | FCES | N | Technical cleaning | 18:32 |
| 18:00 | VAC | Travis | LVEA | N | Checking on cabling | 18:41 |
| 18:32 | FAC | Nellie | MX | N | Technical cleaning | 19:01 |
| 18:45 | VAC | Travis, Jordan | LVEA | N | Starting HAM1 pumpdown | 19:22 |
| 19:26 | CDS | Daniel | LVEA | N | Electronics checks | 19:44 |
| 20:13 | SUS | Fil | LVEA | N | PM1 and RM electronics swaps | 22:00 |
| 20:42 | VAC | Travis | LVEA | N | Craning leak checker, checking pumpdown | 23:19 |
| 20:58 | VAC | Jordan | LVEA | N | Joining Travis | 23:19 |
| 20:58 | CAL | Tony | PCalLab | Local | 21:22 | |
| 22:15 | VAC | Janos | LVEA | N | Joining VAC team | 22:27 |
| 22:30 | CDS | Jackie | CER | N | Retrieving tools | 23:30 |
| 22:32 | VAC | Janos | MX | N | Measurements | 22:54 |
| 22:36 | SEI | Jim, Tony | LVEA | N | HAM2 L4C repair | Ongoing |
| 22:48 | ISC | Daniel | LVEA | N | ISC rack electronics | 23:19 |
Today we installed the missing dual demod chassis in ISC-R2, and Fil pulled the controls cable for the delay line. I then tested that the delay line can be controlled through EPICS and that there are readbacks from the demod chassis.
This complets the hardware installation for the LSC-REFL_B chain.
Gerardo, Jordan, Travis, Randy, Janos Today between 10 am and 2 pm both the Y+ and Y- HAM1 doors have been installed. The installation went smoothly, no issues were found. A valve on the new interlock gauge-assembly was opened, and the HAM1 volume will be purged overnight. This is to clean the in-vacuum surfaces before pumpdown, resulting in lower gas load and therefore faster pumpdown. After the door installation, the pumpdown of the joint HAM1/HAM2 Annulus volumes have been started. 2 turbo/pumping cart systems are pumping the volume, they are standing next to HAM1 and HAM2 - see attached photo. Please do not move the pumping carts, as it can result in immediate damage in the turbo pumps. If you still need to do any activities which need these pumping carts to be moved, please talk to vacuum team members first!