anthony.sanchez@LIGO.ORG - posted 12:49, Tuesday 29 July 2025 (86068)
LVEA Sweep
LVEA has been swept.
LVEA has been swept.
Closes WP 12719. Patrick T., Gerardo M., Dave B. "Update the h0vacly Beckhoff TwinCAT 3 system manager and PLC code to add the IPFCC6 and IPFCC8 filter cavity tube ion pumps and the PTCC7 filter cavity tube gauge. Will require a restart of the PLC and IOC. Will require a DAQ restart." This has been completed. The code on h0vacly is now at commit 6a6e7bc55cb87bc9188312eee1c9a8129bdf7946. No issues. Dave did a burt restore. A DAQ restart is pending.
Trying to narrow down why TMS x is involved in lock losses we have replaced the TMS coil driver that works on F1,2,3 and LF. Chassis S1102670 was replaced with S1102666. The operator returned the system to damping. This is a wait and see test.
Tue Jul 29 10:08:48 2025 INFO: Fill completed in 8min 44secs
After Richard swapped the TMSX coil driver chassis I took a look at the raw OSEM counts for the TMSX (F1, F2, F3, LF) comparing to a time earlier when TMSX and ISC_LOCK were both in the same state (ALIGNED, DOWN). I see that LFs counts are ~2500 counts lower post swap. I'm not sure if there are any other checks to run.
The dolphin IX switch at EX had broken management interface which prevented fencing of front end from the dolphin network for safe reboots.
I installed a new switch. As part of the process, h1cdsrfm was turned off to avoid crashing it and in turn crashing CS and EY dolphin networks.
When I turned h1cdsrfm back on, the cdsrfm port on the switch was already enabled, which led to timing glitches on h1susex and h1iscex. This could have been prevented by re-fencing h1cdsrfm after the new switch was turned on.
I restarted the models on h1susex and h1iscex. A few minutes later, h1susex crashed in the dolphin driver and had to be rebooted. The crash was reminiscent of several susex (plus all of ey) crashes related to restarts of dis_networkmgr on the bootserver. See 67389 for one example.
Switch sn ISX600-HN-000224 was replaces with ISX600-HN-000258.
As part of maintenance we found the cdsmonitor computer had locked up. It came back after a hard boot.
TITLE: 07/29 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 0mph Gusts, 0mph 3min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.05 μm/s
QUICK SUMMARY:
When the SUS_CHARGE was finishing up and SWAPPING_BACK_TO_ETMX ETMX L3 started to constantly saturate and there was a big line on DARM at ~150. This behavior persisted even after the GRD was done and went back to waiting. (tagging SUS).
15:02 UTC I induced a lockloss at by flipping the polarity of H1:IMC-REFL_SERVO_IN1POL, to break the lock and to stop the EX saturations.
I took a look at why there were saturations, and I think it's because of our changes yesterday. When the measurement finished, the ETMX bias was reverted to the new value that Sheila and I used yesterday, but the L3 drivealign gain was set to the old value of 198 instead of the new value of 88. I looked at the script and I don't understand exactly how that happened since it appears that it pulls the values right from LSCparams, but the ndscope I attached shows this is what happened.
I think this was due to the fact that the SUS_CHARGE GRD was not reloaded after lscparams was changed, I've reloaded the GRD.
Workstations were updated and rebooted. This was an OS packages update. Conda packages were not updated.
TITLE: 07/28 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
Today's handoff was recovery (basically waiting a few hours) for the earth to calm down after M6.5 & 7.0 EQs, but there was also Monday commissioning.
For locking, once we were back to "CALM", ran an uneventful alignment, and proceeded locking. The violins were a little elevated, but we've seen worse and after about 20min, H1 went to NLN, BUT only stayed locked a few minutes (no obvious culprits/symptoms on this lock). For the 2nd lock, did NOT run an alignment (but went to Check Mich Fringes +PRMI as per usual), this time H1 did not need to wait extra for violins, BUT in the 1-2 minutes prior to completing the NLN state, there were 0.36 Hz oscillations in ASC (biggest on yaw channels) and then there was a lockloss 30sec after NLN.
Operator Note From the Weekend: All of this is different H1 behavior from the weekend shifts---H1 locking has been easy and automated for the most part with no issues and decent durations on the locks, too! So these 2 locklosses in a row are new/different.
Perhaps these were flukes. Plan is to continue as usual for the night and hope we have better luck locking tonight and look at these issues in the morning.
H1 automatically moved on while I have been writing alogs and texting/chatting with people, and for this 3rd lock of the shift is currently going to 60W as get ready to leave the site.
On a Wildlife note: At sunset, spotted a couple porcupine (hadn't seen them over the weekend) as well as a coyote nearby.
LOG:
HEADLINE: 2nd Lockloss In A Row (Both locklosses tonight within seconds/minutes of completing NLN)--This is also after EXCELLENT/automated locking since Fri, BUT post-commissioning + big EQs today)
For this 2nd locking attempt, H1 didn't linger as long to complete OMC Whitening (because violins were probably better than they were for that last lock). But for this one, as soon as H1 completed NLN, there was ~2min of oscillations (not really ring-ups, mor like glitch, ring down and then glitch and ring down....see attached---where the images 1-4 are screenshots of FOMs).
Had been chatting with Elenna, because she proactively let me know we'd have some SDF diffs from the ETMX changes from Commissioning which preceded the big EQs. After the 2nd Lockloss, I contacted Elenna again. She mentioned the TMSx being an issue. She also asked me what was the frequency of the ASC oscillations it is 0.36Hz (5th attachement).
Elenna looked at the 2nd lockloss and mentioned it was a TMSx thing (https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1437792496). For this 2nd lockloss the ASC oscillations were also more in Yaw (and she mentioned this was something they would be working on tomorrow).
After chatting with Elenna, consulted with Jenne to see about plan for the night and it has been decided to proceed as normal. We only have these 2-data points for locking post-EQ in the last couple hours. So the thought is for the next locks we might get lucky (only fishy thing is locking has been marvelous as far as the last 3 days over the weekend).
Have also contacted TJ to give him a heads up. If same locking issues continue two or more times, plan is to address this problem during Maintenance in the morning.
WP 12696
ECR E2400330
Drawing D0901284-v5
Modified List T2500232
The following SUS SAT Amps were upgraded per ECR E2400330. Modification improves the whitening stage to reduce ADC noise from 0.05 to 10 Hz. The EY PUM and UIM SAT Amps were NOT upgraded.
| Suspension | Old | New | OSEM |
| ETMY MO | S1100098 | S1100088 | F1F2F3SD |
| ETMY MO/RO | S1100079 | S1100083 | RTLF/RTLF |
| ETMY RO | S1100087 | S1000281 | F1F2F3SD |
| TMSY | S1100172 | S1100148 | F1F2F3LF |
| TMSY | S1100107 | S1100172 | RTSD |
| MC1 | S1100128 | S1100118 | T1T2T3LF |
| MC1/MC3 | S1000292 | S1000287 | RTSD/T1T2 |
| MC3 | S1000297 | S1100119 | T3LFRTSD |
F. Clara, J. Kissel, O. Patane, M. Pirello
Once the new satamps were installed, I ran the script satampswap_bestpossible_filterupdate_ECR_E2400330.py to update the compensation filters for these suspensions. These 'best possible' compensation gains come from the tests Jeff did on each satamp before installation, which are found in /ligo/svncommon/SusSVN/sus/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Results/.
My input and the corresponding output is below:
oli.patane@cdsws27:/ligo/svncommon/SusSVN/sus/trunk/Common/PythonTools$ py satampswap_bestpossible_filterupdate_ECR_E2400330.py -o TMSY ETMY_M0_R0
All updated filters grabbed for TMSY
TMSY M1 F1 compensation filter updated to zpk([5.3],[0.0969],1,"n")
TMSY M1 F2 compensation filter updated to zpk([5.28],[0.0964],1,"n")
TMSY M1 F3 compensation filter updated to zpk([5.2],[0.095],1,"n")
TMSY M1 LF compensation filter updated to zpk([5.26],[0.096],1,"n")
TMSY M1 RT compensation filter updated to zpk([5.17],[0.0945],1,"n")
TMSY M1 SD compensation filter updated to zpk([5.26],[0.0961],1,"n")
write /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSTMSY.txt
Done writing updated filters for TMSY
All updated filters grabbed for ETMY
ETMY R0 F1 compensation filter updated to zpk([5.2],[0.0951],1,"n")
ETMY R0 F2 compensation filter updated to zpk([5.2],[0.0951],1,"n")
ETMY R0 F3 compensation filter updated to zpk([5.25],[0.0959],1,"n")
ETMY R0 SD compensation filter updated to zpk([5.35],[0.098],1,"n")
ETMY M0 F1 compensation filter updated to zpk([5.31],[0.0971],1,"n")
ETMY M0 F2 compensation filter updated to zpk([5.27],[0.0965],1,"n")
ETMY M0 F3 compensation filter updated to zpk([5.22],[0.0955],1,"n")
ETMY M0 SD compensation filter updated to zpk([5.17],[0.0946],1,"n")
ETMY M0 LF compensation filter updated to zpk([5.2],[0.0951],1,"n")
ETMY M0 RT compensation filter updated to zpk([5.28],[0.0965],1,"n")
ETMY R0 LF compensation filter updated to zpk([5.29],[0.0967],1,"n")
ETMY R0 RT compensation filter updated to zpk([5.26],[0.0962],1,"n")
write /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSETMY.txt
Done writing updated filters for ETMY
All done! Remember to double check and load in the filters for ['TMSY', 'ETMY_M0_R0']
oli.patane@cdsws27:/ligo/svncommon/SusSVN/sus/trunk/Common/PythonTools$ py satampswap_bestpossible_filterupdate_ECR_E2400330.py -o MC1 MC3
All updated filters grabbed for MC1
MC1 M1 RT compensation filter updated to zpk([5.13],[0.0937],1,"n")
MC1 M1 SD compensation filter updated to zpk([5.25],[0.096],1,"n")
MC1 M1 T1 compensation filter updated to zpk([5.26],[0.0962],1,"n")
MC1 M1 T2 compensation filter updated to zpk([5.18],[0.0947],1,"n")
MC1 M1 T3 compensation filter updated to zpk([5.32],[0.0972],1,"n")
MC1 M1 LF compensation filter updated to zpk([5.12],[0.0938],1,"n")
write /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSMC1.txt
Done writing updated filters for MC1
All updated filters grabbed for MC3
MC3 M1 T3 compensation filter updated to zpk([5.32],[0.0972],1,"n")
MC3 M1 LF compensation filter updated to zpk([5.19],[0.0949],1,"n")
MC3 M1 RT compensation filter updated to zpk([5.35],[0.0979],1,"n")
MC3 M1 SD compensation filter updated to zpk([5.19],[0.0949],1,"n")
MC3 M1 T1 compensation filter updated to zpk([5.31],[0.097],1,"n")
MC3 M1 T2 compensation filter updated to zpk([5.24],[0.0958],1,"n")
write /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSMC3.txt
Done writing updated filters for MC3
All done! Remember to double check and load in the filters for ['MC1', 'MC3']
After this I loaded in these new filters.
The serial numbers in Fil's and OLD and NEW columns are flip flopped in the main aLOG, LHO:85922. Here's the corrected version with the serial number's columns flipped to reflect reality. Suspension Old New OSEM ETMY MO S1100088 S1100098 F1F2F3SD ETMY MO/RO S1100083 S1100079 RTLF/RTLF ETMY RO S1000281 S1100087 F1F2F3SD TMSY S1100148 S1100172 F1F2F3LF TMSY S1100172 S1100107 RTSD MC1 S1100118 S1100128 T1T2T3LF MC1/MC3 S1000287 S1000292 RTSD/T1T2 MC3 S1100119 S1000297 T3LFRTSD
Here's the characterization data and fit results for S1100098 , assigned to ETMY M0's F1F2F3SD OSEMs (Fil refers to this as ETMY MO F1F2F3SD above). The data was taken per methods described in T080062-v3. The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ plotresponse_S1100098_ETMY_M0_F1F2F3SD_20250717.m Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are Optic Stage Serial_Number Channel_Number OSEM_Name Zero_Pole_Hz R_TIA_kOhm Foton_Design ETMY M0 S1100098 CH1 F1 0.0971:5.31 120 zpk([5.31],[0.0971],1,"n") CH2 F2 0.0965:5.27 120 zpk([5.27],[0.0965],1,"n") CH3 F3 0.0955:5.22 120 zpk([5.22],[0.0955],1,"n") CH4 SD 0.0946:5.17 120 zpk([5.17],[0.0946],1,"n") The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ As LHO:85626 and the above LHO:86028 discusses, R_TIA_kOhm is the default 120 kOhm, as it's not used in the compensation filter -- but also because the magnitude of the measurements didn't need me to adjust them; I was able to get a good phase and magnitude fit by just adjusting the zero frequency.
Here's the characterization data and fit results for S1100079 , assigned to ETMY M0/R0's LFRT/LFRT OSEMs (Fil refers to this as ETMY MO/RO RTLF/RTLF above -- note his typo in channel order). The data was taken per methods described in T080062-v3. The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ plotresponse_S1100079_ETMY_M0R0_LFRTLFRT_20250717.m Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are Optic Stage Serial_Number Channel_Number OSEM_Name Zero_Pole_Hz R_TIA_kOhm Foton_Design ETMY M0 S1100079 CH1 LF 0.0951:5.20 120 zpk([5.20],[0.0951],1,"n") M0 CH2 RT 0.0965:5.28 120 zpk([5.28],[0.0965],1,"n") R0 CH3 LF 0.0967:5.29 120 zpk([5.29],[0.0967],1,"n") R0 CH4 RT 0.0962:5.26 120 zpk([5.26],[0.0962],1,"n") The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ As LHO:85626 and the above LHO:86028 discusses, R_TIA_kOhm is the default 120 kOhm, as it's not used in the compensation filter -- but also because the magnitude of the measurements didn't need me to adjust them; I was able to get a good phase and magnitude fit by just adjusting the zero frequency.
Here's the characterization data and fit results for S1100087 , assigned to ETMY R0's F1F2F3SD OSEMs (Fil refers to this as ETMY RO F1F2F3SD above). The data was taken per methods described in T080062-v3. The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ plotresponse_S1100087_ETMY_R0_F1F2F3SD_20250717.m Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are Optic Stage Serial_Number Channel_Number OSEM_Name Zero_Pole_Hz R_TIA_kOhm Foton_Design ETMY R0 S1100087 CH1 F1 0.0951:5.20 120 zpk([5.20],[0.0951],1,"n") CH2 F2 0.0951:5.20 120 zpk([5.20],[0.0951],1,"n") CH3 F3 0.0959:5.25 120 zpk([5.25],[0.0959],1,"n") CH4 SD 0.0980:5.35 120 zpk([5.35],[0.0980],1,"n") The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ As LHO:85626 and the above LHO:86028 discusses, R_TIA_kOhm is the default 120 kOhm, as it's not used in the compensation filter -- but also because the magnitude of the measurements didn't need me to adjust them; I was able to get a good phase and magnitude fit by just adjusting the zero frequency.
Here's the characterization data and fit results for S1100172 , assigned to TMSY M1's F1F2F3LF OSEMs (Fil refers to this as TMSY F1F2F3LF above). The data was taken per methods described in T080062-v3. The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ plotresponse_S1100172_TMSY_M1_F1F2F3LF_20250717.m Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are Optic Stage Serial_Number Channel_Number OSEM_Name Zero_Pole_Hz R_TIA_kOhm Foton_Design TMSY M1 S1100172 CH1 F1 0.0969:5.30 120 zpk([5.30],[0.0969],1,"n") CH2 F2 0.0964:5.28 120 zpk([5.28],[0.0964],1,"n") CH3 F3 0.0950:5.20 120 zpk([5.20],[0.0950],1,"n") CH4 LF 0.0960:5.26 120 zpk([5.26],[0.0960],1,"n") The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ As LHO:85626 and the above LHO:86028 discusses, R_TIA_kOhm is the default 120 kOhm.
Here's the characterization data and fit results for S1100107 , assigned to TMSY M1's RTSDxxxx OSEMs (Fil refers to this as TMSY RTSD above). The data was taken per methods described in T080062-v3. The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ plotresponse_S1100107_TMSY_M1_RTSDxxxx_20250717.m Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are Optic Stage Serial_Number Channel_Number OSEM_Name Zero_Pole_Hz R_TIA_kOhm Foton_Design TMSY M1 S1100107 CH1 RT 0.0945:5.17 120 zpk([5.17],[0.0945],1,"n") CH2 SD 0.0961:5.26 120 zpk([5.26],[0.0961],1,"n") CH3 xx 0.0956:5.23 120 zpk([5.23],[0.0956],1,"n") CH4 xx 0.0957:5.24 120 zpk([5.24],[0.0957],1,"n") The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ As LHO:85626 and the above LHO:86028 discusses, R_TIA_kOhm is the default 120 kOhm.
Here's the characterization data and fit results for S1100128 , assigned to MC1 M1's T1T2T3LF OSEMs (Fil refers to this as MC1 T1T2T3LF above). The data was taken per methods described in T080062-v3. The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ plotresponse_S1100128_MC1_M1_T1T2T3LF_20250717.m Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are Optic Stage Serial_Number Channel_Number OSEM_Name Zero_Pole_Hz R_TIA_kOhm Foton_Design MC1 M1 S1100128 CH1 T1 0.0962:5.26 120 zpk([5.26],[0.0962],1,"n") CH2 T2 0.0947:5.18 120 zpk([5.18],[0.0947],1,"n") CH3 T3 0.0972:5.32 120 zpk([5.32],[0.0972],1,"n") CH4 LF 0.0938:5.12 120 zpk([5.12],[0.0938],1,"n") The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ As LHO:85626 and the above LHO:86028 discusses, R_TIA_kOhm is the default 120 kOhm.
Here's the characterization data and fit results for S1000292 , assigned to MC1/MC3 M1's RTSD/T1T2 OSEMs (Fil refers to this as MC1/MC3 RTSD/T1T2 above). The data was taken per methods described in T080062-v3. The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ plotresponse_S1000292_MC1MC3_M1_RTSDT1T2_20250717.m Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are MC1 M1 S1000292 CH1 RT 0.0937:5.13 120 zpk([5.13],[0.0937],1,"n") MC1 M1 CH2 SD 0.0960:5.25 120 zpk([5.25],[0.0960],1,"n") MC3 M1 CH3 T1 0.0970:5.31 120 zpk([5.31],[0.0970],1,"n") MC3 M1 CH4 T2 0.0958:5.24 120 zpk([5.24],[0.0958],1,"n") The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ As LHO:85626 and the above LHO:86028 discusses, R_TIA_kOhm is the default 120 kOhm.
Here's the characterization data and fit results for S1000297 , assigned to MC3 M1's T3LFRTSD OSEMs (Fil refers to this as MC3 T3LFRTSD above). The data was taken per methods described in T080062-v3. The data was processed and fit using ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ plotresponse_S1000297_MC3_M1_T3LFRTSD_20250721.m Explicitly, the fit to the whitening stage zero and pole, the transimpedance feedback resistor, and foton design string are MC3 M1 S1000297 CH1 T3 0.0972:5.32 120 zpk([5.32],[0.0972],1,"n") CH2 LF 0.0949:5.19 120 zpk([5.19],[0.0949],1,"n") CH3 RT 0.0979:5.35 120 zpk([5.35],[0.0979],1,"n") CH4 SD 0.0949:5.19 120 zpk([5.19],[0.0949],1,"n") The attached plot and machine readable .txt file version of the above table are also found in ${SusSVN}/trunk/electronicstesting/lho_electronics_testing/satamp/ECR_E2400330/Scripts/ As LHO:85626 and the above LHO:86028 discusses, R_TIA_kOhm is the default 120 kOhm.
