TITLE: 09/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY:
LOG:
R. Short, J. Oberling, M. Nakano (remote)
This morning Masayuki ran an assessment of the JAC mode matching after our EOM swap yesterday and came up with new lens positions for the mobile JAC mode matching lenses (IO_MB_L1 and IO_MB_L2). The result we were given:
SITE INSTRUCTION -- two-lens solution (IO_MB_L1 + IO_MB_L2):move IO_MB_L1 by 2.0 mm DOWNSTREAM (toward JAC) (dz_L1 = +2.0 mm)move IO_MB_L2 by 34.6 mm UPSTREAM (toward the PMC) (dz_L2 = -34.6 mm)(decomposition: common -16.3 mm = waist position, differential +18.3 mm = waist size)L1-L2 gap: 60 mm -> 23 mm
Moving IO_MB_L1 went smoothly with no change to the beam alignment. However, we were not able to move IO_MB_L2 the full requested amount due to interference between the lens mounts. Instead of the requested 23mm separation between the lenses, the best we could get was ~26.5 mm. To get any better than this we would have to completely change the mounting setup for the lenses. After moving IO_MB_L2 we had to yaw it a little to recover the beam alignment onto our irises. While the irises looked good in the PSL enclosure, when we tried to lock JAC it wouldn't lock; Masayuki had to do some alignment tweaks to get the JAC to lock. This isn't entirely surprising given that our alignment irises are several meters away from the JAC in HAM1, so what looks good on the irises isn't necessarily good enough to lock the cavity. Once the cavity was locked, Masayuki began tuning things up and we left the enclosure. He's running his mode matching assessment code now, so we'll see what the results show (preliminary report: "The mode matching seems better." -M. Nakano).
Randy, Tony, Travis, Gerardo, Jordan
Once we got the go ahead for re-installing the BSC door, Randy, Tony and I re-installed the +Y door without any issues. There was some particulate/viton pieces at 12 and 3 o'clock on the flange, but we were able to remove them using IPA wipes.
Bolts were torqued, then Travis and Gerardo set up the turbo and aux cart to pump on the annulus volume. Annulus pumping in progress.
While looking at the ETMX for it's rubbing issue of late, we have been doing some data mining to understand why the ETMx Main and Reaction chain vertical BOSEMs (M0 and R0 LF and RT) are not where I would have assumed to leave them when we installed the "new" ETM test mass and rebuilt the suspension in 2018. Today, these vertical sensors are all ~6-8k counts when they should be closer to 11k per the open light voltage counts we determined in 2018 alog 41937.
I worried about temp and slow drift which could be also contributing to the rubbing, but in fact, it seems like something happened immediately at closeout in 2018 that put these vertical sensors off the nominal centered 11k value. (sigh)
In the end, hunting for an event or evolution which could have lead to the L1 LL rubbing of late has not been fruitful.
Anywho, since I waded through the deep water of everything-ETMX-SUS via temp change hunts, etc, I will post it here for the next time we wonder about what the longer history of the EX VEA temp has been doing:
Dec 2014 - CR excursions turned back down from 68 to 65
Jan 6, 2015 - 15883 - John turns temp “back to 65F (down from 67F)” - likely after a vent?
2015-2016 - No one writes actual temp values just things like “seasonal change so I turned a heater on”
May 2017 - 36271 - Upgrade to HVAC controls everywhere, EX week of ~May 15-18th, Kissel alog on calibration discrepancy 64 is actually 68…
May 22, 2017 - 36331 - Jeff/Bubba request to increase from 64 to 66 degC
May 30, 2017 - 36536 - Bubba changes which sensors are in-loop at EX
June 14, 2017 - 36894 - Bubba changes which sensors are in-loop at EX
Jan 23, 2018 - 40233 - Bubba/Hugh banter about where sensors in EX are mounted
Feb-Jun 2018 - Vented, warm with Cleanrooms on, new ETM/AERM installed
________________________________________
June 1, 2018 - 42283 - Chamber closeout TFs ETMx
Vertical sensors get “bumped” or set from nom 11k to ~8k somehow.
June 6, 2018 - 42351 - Bubba puts temp back down to 65degF after chamber closeout
June 26, 2018 - 42664, 42686 - Chandra asks to put temp up to 68defF to aid in outgassing/pumpdown, Kissel/Chandra banter about temp increase. Chandra states that 65degF is more energy efficient than 68degF
June 29, 2018 - 42722 - Bubba puts temp back down to 65degF
Jan 27, 2022 - 61472 - Bubba/Fil move some sensors/cables, “The temperature at the end stations by request of Jeff Kissel several years ago was maintained at 65 degrees F.” - Probably need to go back to fact check this since originally Kissel suggested 66 per logs found above. 0.5degF change but Vert’s see -2k OLV change, weird.
Mar16 ’23-Jun 22 - EX "Rubbing" and 11Hz issues, offsets put on, then later turned off because not an issue “anymore” (68034, 68121, 63766) - So, in hind sight, could have had the L1 LL flag close and maybe brushing in 2022 then it kinda alleviated itself.
Aug 28, 2023 - 72497 - Kissel nice blog analysis of temp on EY, FWIW
Sept 14-15, 2026 - Rubbing issues on ETMX - L1 LL flag brushing. Vented, moved LL BOSEM over away from flag. See that ETM M0/R0 V BOSEMs have been at ~6-8k instead of nominal ~15k for many many many years (SUS is “UP” which correlates to “COLD”).
Sept 15, 2026 - 91912 - Richard raising EX from 65F to 67F as read out by HVAC chans. (This should drop the suspension and more center the V BOSEMs, but likely not an actual problem.)
After various ETMX transfer functions and offset tests, it looks like ETMX isn't rubbing anymore!
I did these tests this morning and wasn't able to get any rubbing. For everthing I did, L1 LL (and everyone else) moved as expected and no one got stuck anywhere. The offset tests that I did involved putting offsets in the L1 COILOUTF banks, M0 TEST banks, and R0 TEST banks and making sure L1 osems were all responding how we expected them to.
We noticed that L1 UL has been a little more jaggedly compared to the other L1 osems since we went to air, but the osem spectra looked good (comparing to its brethren) and putting a drive in M0 to move L1 around shows that UL is moving and behaving normally. We think it's probably just related to its current flag vs osem position as we put air in the chamber.
Transfer Functions
Settings:
- HEALTH_CHECK
- OPTICALIGN OFFSETS ON
- DAMP OFF
- ISI in ISI_DAMPED_HEPI_OFFLINE
M0
Data:
/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGM0/Data/2026-09-15_1540_H1SUSETMX_M0_WhiteNoise_{L,P,Y}_0p02to50Hz.xml
/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGM0/Data/2026-09-15_1540_H1SUSETMX_M0_WhiteNoise_{T,V,R}_0p03to50Hz.xml
r13152
Results:
/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGM0/Results/SAGM0/Results/2026-09-15_1540_H1SUSETMX_M0_ALL_TFs.pdf
r13153
R0
Data:
/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGR0/Data/2026-09-15_1615_H1SUSETMX_R0_WhiteNoise_{L,T,V,R,P,Y}_0p03to50Hz.xml
r13148
Results:
/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGR0/Results/2026-09-15_1615_H1SUSETMX_R0_ALL_TFs.pdf
r13150
Spectra:
L1
Data:
/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGL1/Data/2026-09-15_H1SUSETMX_L1_OSEMINF_spectra_invac_vs_inair.xml
r13149
Please do not make any changes to real time models with simulink or foton until after the RCG upgrade.
Quickly checked the ugf of the JAC servo. At a filter gain of 0.8, the nominal gain was 300Hz. This seems the same as before, indicating the modulation index didn't change much with the EOM swap. The attached plot shows the ugf at 400Hz with a gain of 1.0.
Reminder: we attenuated the power by half to the IMC refl diode and IMC WFS after a power outage during O4, 86884, using a half wave plate along the refl path in IOT2L.
The attenuation was such that we maintained the desired power on IMC refl and the WFS by reducing the refl path power by 2, 86989, in other words, whatever the power outage did to the upstream mode matching, it was such that the power reflected off the IMC when locked had doubled.
To summarize, our nominal power to IMC refl when locked with ~60 W input is 18.5 mW. The WFS nominal is 0.95 and 0.76 mW respectively.
Now we have JAC, so I checked the powers at IMC REFL. Looks like the JAC team turned up the input power to the IFO to 60 W recently. For 60 W input with IMC locked (guardian state 100), we now have 1.57 mW on IMC REFL, 0.079 and 0.064 mW on WFS A and B. I think this indicates we should adjust the half wave plate to increase power to IMC REFL. I'm not sure if we can get back to nominal since the powers are more than an order of magnitude lower than before, but we should be able to at least double it.
Reminder that the "IMC PWR IN" diode is actually reporting the power into JAC. We have a JAC trans diode, but it still has a funny calibration, so I did not include it here because the number it says is totally wrong.
Betsy, Oli, Tony, Camilla
Yesterday when Betsy, Oli and Tony were inside the ETMX BSC (WBSC9) they took some photos of the transmon mirrors,
Reason for this is that we've had issues with both the ALS 91306 and HWS 91248 beams at EX.
Full telescope assembly is D1002460. The optics on the transmon table are in D1201457. The mirrors are called from transmon towards ETM: M2 , F2, F1, M1. D1102361 and Madi's cartoon, attached from G2500646 shows this.
Betsy notes that the optics with the side HR facing -X (F2 and M1) appear to have considerably more dust on the those facing +X. Is purge air or similar causing this? The dusty side is direction ETMX faces. All the mirrors coating looks intact.
The HWS path green beam made it though the transmon and retroreflected off ETMX and back onto the HWS CCD, but the alignment isn't perfect as the return beam looks clipped, photo.
PicketFence was updated to the latest version and restarted.
IOC is not updating server status. Results are being updated, so the PicketFence is talking to the IOC.
Picket Fence fixed and working. Display now shows latency for each station.
Richard, Betsy, Eric.
Looking back many years we can not find a reason for the temperature setting of 65 Deg F at EX. Today I raised the temperature to 67 Deg F. Wtih the clean room on we have seen a diurnal swing of 3 Deg F with no effect on the suspension so this rise should not have a negative impact on the suspension.(sagging) It may actually help.
However, It will probably change the balance of the BRS. Jim W is aware
[Jenne, Elenna]
As a reminder, Jenne had engaged a "slow let go" servo for beamsplitter pitch during O4 because of some possible heating transient that affected relocking, see 85002.
Now we have the BBSS, so we don't expect to see this issue. I undid the steps that Jenne did in her alog above to revert the settings:
ISC_DRMI is loaded and I checked the code to the svn.
Richard, Jonathan, EJ, Erik, Fil, Dave:
First thing this morning we did a round of h1seih16 power cycling, with Richard power cycling the IO Chassis. The models ran for about 5 minutes this time and then ADC1-timed-out like before.
Then we power cycled the CPU only, and this time the IOP would not run because of a card count issue. And indeed, both showcards and lscpi report that the A2-slot4 is empty, which contains the second 16bit-DAC card.
So we are operating under the theory that all the recent problems have been caused by a flakey DAC1, manifesting as ADC1 timing errors. We are preparing to replace DAC1.
Card 2
Extender 1
slot 1: (10ee d8c6) Xilinx Corporation,LIGO Timing Board
slot 2: empty
slot 3: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
slot 4: (10b5 3120) PLX Technology, Inc.,General Standards 16AO16
Card 3
Extender 2
slot 1: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
slot 2: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
slot 3: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
slot 4: empty <<< Should be 16AO16 DAC1 <<<<
Card 4
Extender 3
slot 1: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
slot 2: (10b5 3101) PLX Technology, Inc.,General Standards 16AI64
slot 3: empty
slot 4: empty
Card 5
Extender 4
slot 1: (1221 8682) Contec Co., Ltd,Digital I/O DIO-64
slot 2: (1221 8682) Contec Co., Ltd,Digital I/O DIO-64
slot 3: empty
slot 4: empty
Opened WP13616 to cover this work.
Fil is replacing the second 16bit-DAC.
second 16bit-DAC:
| old card (removed) | 100922-16 |
| new card (installed) | 101118-18 |
At time of writing, h1seih16 has been running for 30mins.
Camilla and I had another look at the filter cavity alignment this afternoon. We see green flashes of 7.1 counts, where we would expect about 30 counts based on the launch power and O4 FC transmission. These slider values are in the attached screenshot. Went to SQZT7 to align the beam onto FCGS refl diode, didn't see any signal in the DC readback of that diode. However, now there are mini locks of the filter cavity on the 00 mode, so this looks like the diode is working.
Jennie W, Jason O, Ryan S,
Summary: PSL EOM swapped, alignment good enough to lock JAC but the mode-matching to JAC looks different as the JAC REFL camera spot has changed.
We entred the PSL enclosure and turned down power through EOM to less than 2W at the manual waveplate.
We then recentred the first iris in the MB path after the EOM in the vertical direction as the beam was slightly too low. The second iris was centred ok.
Then we closed shutter 2 and unplugged the 118 and 9 MHz cables.
We terminated the 45 MHz cable as this is the only one still plugged into an active voltage source.
We had to remove the EOM along with the white spacer it sits on to undo the screws on the bottom of this spacer that go into the EOM unit.
After swapping the new EOM onto the spacer we screwed it back down on the mount.
The beam was now off of the first iris in both pitch and yaw. It was too low and slightly in the +y direction.
To fix this in the y direction Jason adjusted IO_MB_M2 in yaw. The beam was not making it through the the EOM aperture. We loosened the side screws on white spacer and used slop in the large screws slots which hold the spacer on the bottom mount to yaw the eom and fix this . Then we used the pitch of the IO_MB_M2 mirror to centre the beam in the vertical dirtection on the first iris.
After tightening back the screws on the mount and spacer we took a calibration sweep of the rotation stage to re-calibrate the output power to the interferometer.
Then we took the power guardian to 2W and re-locked the JAC. The relfected beam from the locked JAC has changed shape on the camera so we assume the mode-matching is different.
Masayuki will measure the mode-matching and we will improve this with lens moves on the PSL table tomorrow.
On the rotation stage calibration mentioned above, we started by closing the shutter to not allow any light into HAM1 (to be safest since we weren't sure exactly how well IOT1 would take ~100W of power in its current state) and paused the LASER_PWR Guardian while keeping the ISS enabled. I ran the typical calibration script (/userapps/psl/h1/scripts/RotationStage/CalibRotStage.py), updated the calibration factors, then used the Guardian to return to 2W; see SDF screenshot for the changes. It looks like it was October of last year when the last calibration was run, and now that we have high power through the EOM, it doesn't surprise me that the input maximum power is much higher (this time saw up to 92.1W).
The "new" EOM is aLIGO #4, which was operated in the PSL from aLIGO install until the swap to the 4f version in 2018. EOM crystal serial number is 10252005 (unchanged from previous use).
I unplugged some cables to an interface chassis in the CER this afternoon to do some CPS whitening measurements. Not sure what has happened, but when I plugged everything back in, the corner 2 cps didn't come back alive, and are sitting at ~0 cts. I have to leave early today, so I'm out of time to diagnose, but hopefully I can fix this tomorrow morning when I get back on site. This means HAM2 won't be able to isolate, until the corner 2 CPS are working again.
It turns out, the cabling at the HAM2 rack is kind of confusing and I was inadvertently swapping the cables for corner2 and corner1&3. Apparently when we had to swap cables around during install to make room for the PSL, we plugged the cable marked CPS2 into interface chassis 1 and vice versa. Confusingly, we did not do this swap with the power cables, so power cable 1 is coupled with CPS cable 2. Fil says he thinks we can probably straighten this out at the chamber, but we are running out of time today, so we will come back to this. FRS is : https://services1.ligo-la.caltech.edu/FRS/show_bug.cgi?id=39086
Sheila, Elenna, Camilla
Sheila was questioning if something is drifting for us to need an initial alignment after the majority of relocks. Elenna and I noticed that BS PIT moves a lot both while powering up /moving spots and while in NLN. Unsure from the BS alignment inputs plot what's causing this.
This was also happening before the break (see below) but the operators were similarly needing more regular initial alignments before the break too. 1 year ago this was not happening, plot.
These large BS PIT changes began 5th to 6th July 2024 (plot). This is the day shift from the time that the first lock like this happened 5th July 2024 19:26UTC (12:26PT): 78877 at the time we were doing PR2 spot moves. There also was a SUS computer restart 78892 but that appeared to be a day after this started happening.
Sheila, Camilla
This reminded Sheila of when we were heating a SUS in the past and causing the bottom mass to pitch and the ASC to move the top mass to counteract this. Then after lockloss, the bottom mass would slowly go back to it's nominal position.
We do see this on the BS since the PR2 move, see attached (top 2 left plots). See in the green bottom mass oplev trace, when the ASC is turned off on lockloss, the BS moves quickly and then slowly moves again over the next ~30 minutes, do not see simular things on PR3. Attached is the same plot before the PR2 move. And below is a list of other PR2 positions we tried, all the other positions have also made this BS drift. The total PR2 move since the good place is ~3500urad in Yaw.
To avoid this heating and BS drift, we should move back towards a PR2 YAW of closer to 3200. But, we moved PR2 to avoid the spot clipping on the scrapper baffle, e.g. 77631, 80319, 82722, 82641.
I did a bit of alog archaeology to re-remember what we'd done in the past.
To put back the soft turn-off of the BS ASC, I think we need to:
Camilla made the good point that we probably don't want to implement this and then have the first trial of it be overnight. Maybe I'll put it in sometime Monday (when we again have commissioning time), and if we lose lock we can check that it did all the right things.
I've now implemented this soft let-go of BS pit in the ISC_DRMI guardian, and loaded. We'll be able to watch it throughout the day today, including while we're commissioning, so hopefully we'll be able to see it work properly at least once (eg, from a DRMI lockloss).
This 'slow let-go' mode for BS pitch certainly makes the behavior of the BS pit oplev qualitatively different.
In the attached plots, the sharp spike up and decay down behavior around -8 hours is how it had been looking for a long time (as Camilla notes in previous logs in this thread). Around -2 hours we lost lock from NomLowNoise, and while we do get a glitch upon lockloss, the BS doesn't seem to move quite as much, and is mostly flattened out after a shorter amount of time. I also note that this time (-2 hours ago) we didn't need to do an initial alignment (which was done at the -8 hours ago time). However, as Jeff pointed out, we held at DOWN for a while to reconcile SDFs, it's not quite a fair comparison.
We'll see how things go, but there's at least a chance that this will help reduce the need for initial alignments. If needed, we can try to tweak the time constant of the 'soft let-go' to further make the optical lever signal stay more overall flat.
The SUSBS SDF safe.snap file is saved with FM1 off, so that it won't get turned back on in SDF revert. The PREP_PRMI_ASC and PREP_DRMI_ASC states both re-enable FM1 - I may need to go through and ensure it's on for MICH initial alignment.
RyanS, Jenne
We've looked at a couple of times that the BS has been let go of slowly, and it seems like the cooldown time is usually about 17 minutes until it's basically done and at where it wants to be for the next acquisition of DRMI. Attached is one such example.
Alternatively, a day or so ago Tony had to do an initial alignment. On that day, it seemed like the BS took much longer to get to its quiescent spot. I'm not yet sure why the behavior is different sometimes.
Tony is working on taking a look at our average reacquisition time, which will help tell us whether we should make another change to further improve the time it takes to get the BS to where it wants to be for acquisition.
With the new BBSS, Elenna has removed the 'slow let-go' of the pitch top mass. See alog 91910.
See attached for pictures of the lens configuration before and after our change (where the two lenses are much closer together after).