We've asked Tony to try locking the IFO with SR3 and SR2 in these very different positions. With these optic positions, we don't want the AS_C offsets in place that we used to get locked yesterday. So, I have now accepted in safe.snap for the ASC model that the AS_C offsets are off (but I've left the numerical values).
If we're unable to lock, then the plan is to ask Tony to revert SR2 and SR3 and turn back on those offsets. Since the offset switches are part of SDF revert, to turn them on the plan should be to go Sitemap -> ASC -> Overview -> AS_C (bottom left) -> Turn on the offsets and save the diff in SDF.
First attachment is my saving the safe.snap file with the offsets off. Second attachment is where the offset buttons are.
We've gotten as far as PRMI locked (with arms held off resonance with ALS, as usual), and at some point as soon as PRMI ASC comes on MICH Yaw gets a 2.2 Hz wiggle.
EDIT to add that a thing to try could be to turn off the ASC during PRMI (set lines 737 and 738 of ISC_DRMI.py to False). If we have similar problems in DRMI ASC, also set lines 1026-1031 to False. Save, then reload ISC_DRMI guardian.
EDIT EDIT: I just set line 737 to False. It may be beneficial to tune up the alignment by hand.
EDIT EDIT EDIT: We're still getting wobbles, even though the guardian had turned off the MICH ASC, and I had by-hand turned off the PRC1 ASC.
EDIT EDIT EDIT EDIT : I also set PRC1 to False on line 738 and loaded. I have to go now though, so good luck Tony!
We had an almost 30 hour lock, but probably we want PRMI ASC to work again now that we're relocking. RyanS and I have just set lines 737 and 738 back to True to enable PRMI ASC.
LHO control room crew-
We suspect that something about the output Faraday has changed.
The only thing that seems common to these two is the OFI, a baffle in it's vicinity or something like a wire from the fast shutter impacting the beam on the way to OM1. We'd actuated the fast shutter and see no change in the AS camera, so we don't think that the fast shutter is the issue, which points us to looking at the OFI.
Editing to add that Keita points out something like a bad spot on OM1 is also a possibility.
TITLE: 04/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 14mph Gusts, 9mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.09 μm/s
QUICK SUMMARY:
IFO Unlocked and down for Maintenance on potentially the OFI which may be wildly misaligned, as implied by the reduced power on the photo diodes down stream from the OFI.
Feels like an all hands on deck sort of issue.
Mr. Crouch is running TF on the OFI and I'll be taking over Initial Alignment while this is happening.
DriptaB, TonyS, RickS, FranciscoL
On Wednesday, April 24, we updated the EPICS variables,
for X-End OLD --> NEW:
H1:CAL-PCALX_FORCE_COEFF_RHO_R | 10692.3 --> 10716.6
H1:CAL-PCALX_FORCE_COEFF_RHO_T | 8334.52 --> 8305.09
H1:CAL-PCALX_FORCE_COEFF_RX_OPT_EFF_CORR | 0.99460 --> 0.99480
H1:CAL-PCALX_FORCE_COEFF_RX_PD_ADC_BG | 0.25990 --> 0.71360
H1:CAL-PCALX_FORCE_COEFF_TX_OPT_EFF_CORR | 0.99350 --> 0.99380
H1:CAL-PCALX_FORCE_COEFF_TX_PD_ADC_BG | 8.81310 --> 9.65710
H1:CAL-PCALX_XY_COMPARE_CORR_FACT | 0.99880 --> 0.99790
For Y-End OLD --> NEW:
H1:CAL-PCALY_FORCE_COEFF_RHO_R | 10652.0 --> 10649.6
H1:CAL-PCALY_FORCE_COEFF_RHO_T | 7157.80 --> 71456.2
H1:CAL-PCALY_FORCE_COEFF_RX_OPT_EFF_CORR | 0.93100 --> 0.99340
H1:CAL-PCALY_FORCE_COEFF_RX_PD_ADC_BG | -8.15000 --> -0.25910
H1:CAL-PCALY_FORCE_COEFF_TX_OPT_EFF_CORR | 0.92000 --> 0.99230
H1:CAL-PCALY_FORCE_COEFF_TX_PD_ADC_BG | 1.75161 --> 1.82388
H1:CAL-PCALY_XY_COMPARE_CORR_FACT | 1.00150 --> 1.00130
To reflect the most recent calibration of the receiving pcal power sensor (Rx) at both end stations. Screenshots of the MEDM screens show the new values on the green boxes. Attached .txt file shows the caput command used to change the values.
The total chage of -0.16% for X-End, and -0.02% for Y-End for the combined fiducial displacement factors.
We are planning on regularly updating the Pcal calibrations at a cadence of roughly 3 months since over the course of O4a we saw systematic variations on Rx sensor calibration (ct/W) by as much as +/- 0.3% at both sites. The systematic variations at LHO are shown in the 'RxprimeTrendsLHOXandYO4a_wRxTemp.pdf', the top panel for X-end and the bottom one for Y-end. The scatterplot with error bars represent Rx responsivity (units of ct/W) values from pcal end station measurements normalized to the EPICS variable entered for $\rho_R'$, where $\rho$ is the responsivity and prime denotes corrected for optical efficiency, at the beginning of O4a. Values are denoted on the left-hand side of the y-axis. The large dataset represents temperature measurements of the corresponding VEA, and the values are denoted on the right-hand side of the y-axis. These plots suggest that variations on the calibration of the Rx sensor might be due to outside temperature variations, investigations ongoing.
End station measurements allow us to monitor the change in Rx calibration over time. Every end station measurements produce a report that includes a plot in units of ct/W, along with a calibration of the working standard power sensor (WS) made on the same day. We will continue making end station measurements regularly and depending on how much the Rx calibration changes over time, we may update the Pcal EPICS variables.
Sheila, Camilla, Jennie + CR
After Jenne got AS port power back with large SR2/3 yaw offsets in 77388, Sheila wanted to check we could get our SQZ beam back to the power levels we had during the SQZ to OMC scan in 77213 (~75 counts on AS_A and AS_B NSUMs).
We revered ZMs (alignment and PSAMs), SRM, OM1, OM2 to that scan time: April 16th 19:10UTC. But in the old alignment we can only see ~10 on AS_A AS_B NSUM, plot from Sheila attached. So something has changed in the alignment in the SQZ to OMC path too: SQZ, ZM4,5,6, OFI, SRM, OFI, OM1, OM2.
From this old OMC-SQZ scan alignment, by pitching ZM5 200urad (sliders) we could get back to 75 on NSUM plot (AS_AIR beam too low off camera). OR by yawing ZM5 ~150urad (190urad on sliders/ 130urad on osems) we could get back to 70 on NSUMs plot (AS_AIR camera looks better).
Reverting PSAMs back to nominal.
Sheila, Camilla. Relooked at the times from the original SQZ_OMC scan (04/16) to the checked alignment back to this scan last week (04/24): when we moved ZM5 to increase power on AS_C, the alignment onto AS_C gets much worse, plot attached. AS_A/B can;t be trusted as much as the OM3_P is different. Today we translated he beam in yaw to investigate this, alog pending.
We're still working to understand why we've got problems, but we've at least found *an* alignment of SR3 and SR2 that seems to prevent clipping at the AS port when we're centered on AS_C. To get here, I had to move SR3 and SR2 by very large amounts, much more than they ever drift. In order to more accurately see changes in power levels on the AS WFS, I had the DC centering loops engaged in this single bounce off of ITMY configuration.
In the attached plot, the bottom row is the slider values for SR2 yaw and SR3 yaw (so, in microradians). As a reminder, folks have been checking all day and there is no indication from suspension sliders, OSEMs, or where applicable oplevs, that any of our optics have moved nearly this much, so this should not have been necessary. That said, we've often found that (normally at least) we have lots of leeway clipping-wise when chosing SR3 and SR2 positions. There's no reason a priori that I can think of that would prevent us from locking with these SR2 and SR3 positions, but TJ and Camilla point out that having moved SR3 this much may make it challenging to also have the HWS paths aligned. These moves are basically my moving SR3, then moving SR2 to re-center on AS_C.
The top row of the attached plot is the centering on AS_C. So, we would like to only evaluate the amount of power on AS_A or AS_C (middle row) when the beam is centered on AS_C. It turns out that it's the low-ish part of the power curves that are the values when AS_C is centered.
I have highlighted using a blue line on the AS_A curve (middle row right side) roughly the trend. When we started (at about -42 mins) the power on AS_A when AS_C is centered is at about 1.64 units on this y-axis, and as we move SR3 and SR2 in yaw, I can increase the power on AS_A to about 1.85 units. Jennie found that a time of equivalent single bounce configuration from a week ago, we had 1.87 units on this y-axis. (This y-axis is just AS_A_NSUM * 0.001). I didn't take the time to go to the 'other side', but the trend of power on AS_A seems to show that we're into the plateau region, and at the same time the AS AIR camera looked much more normal and unclipped.
EDIT to note that part of the reason it's helpful to wait to evaluate AS_A power until AS_C was centered, is that that also gave time for the DC centering loops to catch up to my big SR3 moves. I think that the reason AS_A sometimes is higher than the blue marker curve, is that the beam was clipping on AS_A while the DC centering loops were catching up.
slider changes:
Sheila, Jenne, Camilla, Jennie, Jim, Rahul, others
We continue to struggle to understand why our alignment in our output arm is suddenly so bad and cannot get back without massive offsets and noise introductions. Jim and Rahul have each looked at their respective subsystems and given the OK. Somehow our alignment have moved in the last 30 hours and we cannot recover it. Here are some summary bullets from the morning:
Sheila, Jennie W
Sheila noticed our range was really high in the lock at 14:51 UTC on the 17th April.
All camera offsets are the same from the good range time (16:19:53 UTC) on the 17th (left cursor) compared to a lock from last night at 2024/04/24 13:20:30 UTC (right cursor).
The offsets for these currently (second cursor) are not in lsc_params and since we had better build-ups on the 17th (first cursor), we should fix these offsets and see if we can optimise A2L gains again in the next commissioning window after the IFO range is recovered.
The best build-ups were immediately after this good time when we changed the PIT2 and YAW3 but this made the range worse.
We looked at this again today and realised the camera offsets had been set back to their former values from before our changes to PIT2 and YAW3 on April 17th when the cameras servo guardian went through state 405 (TURN ON CAMERA FIXED OFFSETS). Then we realised we may have forgotten to reload the camer servo guradian when we made these changes so we took reloaded it then took it to DITHER ON state, then to CAMERA SERVO ON and now we have CAM PIT2 OFFSET = -173 and CAM YAW3 OFFSET = -349.5 as is set in lsc_params.
FIXED ISSUE
Ryan C, Rahul
Rahul and I did a bunch of SUS health checks in which we did not see anything out of the ordinary.
1) I ran the rubbing script (/opt/rtcds/userapps/release/sus/h1/scripts/SUS_TopMassOSEMs.py) comparing to a time last week, no issues.
2) We took OFI, OM1-3 spectra. We compared to spectra on monday, the OMs are experiencing some overflows (H1SUSIFOOUT) but we're not sure why, overall it looks fine.
3) QUADs: Binary I/Os, OSEMs, Voltmons. Trending over the past week didn't reveal anything except ITMX_L1_OSEM UL might be very slowly dying.
4) OPLEV sums
Two lock acquisitions last night do NOT see the 9Hz buzzing at state 557 TRANSISITON _FROM_ETMX seen in 77359.
Plots at 2024/04/24 04:21UTC and 07:32UTC attached. Also check the two lock acquisitions that we lost before NLN before this.
Wed Apr 24 10:09:59 2024 INFO: Fill completed in 9min 56secs
Gerardo confirmed a good fill curbside.
Well pump has been started to replenish the fire water tank. The pump will run for 4 hours.
TITLE: 04/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 0mph Gusts, 0mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY: Still locked but with a range around 120Mpc. Lots of noise in almost the entire of the spectrum.
TJ took no SQZ time this morning from 15:31:30 to 15:45:00UTC. Plot attached comaping now to no SQZ time from April 12th 77133
TJ and I looked at the OFI (plot) and over the last 5 days, the RT voltmons has changed more than usual. We aren't actively damping the OFI though so are unsure if these VOLTMONs are just reading noise...
We checked the HWS _PROBE_TOTAL_PIXEL_VALUE and the position of IFO beam on IX and IY HWS, from last good long range power up 2024/04/21 20:55UTC (1397768177) to last nights power up 2024-04-24 07:26UTC (1397978835). No difference in the two times seen. HWS cares about SR3.
[Jenne, Ibrahim, RyanS, Sheila, Robert, TJ, Elenna, Jennie, Oli, Jim, others]
Something is different with the IFO today, and it's not great. Since we haven't pinpointed what the problem is, we don't really know if it is related to our mediocre range last night and struggles to relock after that, or if it is new from maintenance day. So far, the solution to get back to (almost) NLN has been to put in *massive* offsets in AS_C. This did enable us to get locked (and avoided the ASC ringup that was causing locklosses this afternoon), but it left us with a large scatter shelf in DARM. Robert had the good suggestion that we see if, once locked, we could walk the offsets back to their nominal values of zero; doing this caused an ASC ringup, and it's probably the same thing that we'd been seeing throughout the day. So, indeed going toward the nominal offset of zero for pit and yaw on AS_C is not an okay place right now.
We began to narrow in on AS_C since, during initial alignment, it is in-loop for SR2 and SRY alignments, and it was causing SR2 to be pulled very far in yaw. We were seeing this visually on the AS_AIR camera, which looked really unusually terrible. In the end, Sheila hand-aligned SRY, and then put offsets into AS_C such that the ASC now servos to that point. But, the offsets used are +0.46 in pit and -0.85 in yaw, so really, really big. However, with these in place, we were able to let DRMI ASC run, and it ran fine.
Since that worked (the large AS_C offsets), we let the IFO relock the rest of the way, and it kept on working. Per a suggestion from Elenna from earlier in the afternoon, after we completed REDUCE_RF45 I manual-ed to ADJUST_POWER and increased the power 5W at a time (waiting for ASC to mostly converge between steps) until we were at 60W, then I manual-ed back to LOWNOISE_ASC. After that, I just selected NomLowNoise and guardian finished things. We ended up 'stuck' in OMC_WHITENING, although the violins were coming down with the damping that guardian was doing. It was around here that I tried ramping down the AS_C yaw offset with 10 sec ramp times to see if it would reduce the scatter shelf that we saw in DARM. See first attachment, with the scatter shelf circled. My first step was to -0.80, second step was to -0.70, and we had an ASC ringup and lockloss at this step. I wasn't sure after the first step, but definitely as we were ramping to the second step (before we lost lock) RyanS, Robert, and I all agreed that the scatter shelf was getting worse, not better. But, we lost lock before I could put the offset back.
We still don't understand *why* we need these large offsets in AS_C, just that we do. Since I hadn't yet SDF-ed them, we had 2 locklosses from ENGAGE_DRMI_ASC when the nominal zero-offsets were used in AS_C. I have since saved the AS_C offset values, and the offset switch being on, in both the safe.snap and observe.snap SDF files. The second and third attachments show these.
We've been trying to think through what from maintenance day could possible have had anything to do with this, and the only things we've come up with are the h1asc reboot and the grounding of SUS racks. From Jeff's alog about the h1asc changes, it seems quite clear that those are just additions of monitor points, and that shouldn't have had any effect. While we were unlocked after our one successful attempt, Robert went into the LVEA and un-grounded those two racks that Fil grounded earlier today. Robert is quite sure that he's checked the grounding of them in the past without any consequence to our ability to lock, but just in case we've got them undone for tonight. So far, that does not seem to have changed the fact that we need these crazy offsets in AS_C. Just in case, Robert suggests we consider rebooting h1asc tomorrow to back out other changes that happened today (even though those shouldn't have anything to do with anything).
For right now (as of ~7:20pm), my best plan is to see if we can get to Observing, and should a GW candidate come through, have the detchar and parameter estimation folks exclude all data below ~80 Hz. This is a really very poor solution though, since this huge scatter shelf will be quite detrimental to our data quality and could cause our search pipelines to trigger on this scattering.
Related thought, but not yet thoroughly investigated (at least by me), is whether our problems actually started sometime yesterday or last night, and aren't at all related to maintenance. As partial 'evidence' in this direction, I'll note that kappa_c has been dramatically low for the last two Observing segments last night. We haven't gotten all the way to NLN tonight yet (OMC_WHITENING didn't finish before we lost lock), but kappa_c looks like it might be even lower now than yesterday. The 4th attachment shows kappa_c for last night's locks (screenshot taken before we locked today). So, maybe we've had some clipping or severe degredation of alignment in the last day or so. Jim checked all of the ISIs, and nothing seems suspicious there.
Last thing - since we had run the dark offsets script earlier this afternoon (probably was a red herring) and I saved all of the SDFs in the safe.snap, we will have diffs for Observe.snap. All diffs that have _OFFSET in them should be accepted (and screenshot-ed).
Oh, also, the squezing was not at all good when we locked. I suspect that it's because the squeezer alignment isn't matched to this bad-but-working alignment of the rest of the IFO. Sheila should be able to log on in a little while and can hopefully take a quick look to see if there's anything to do.
Adding screenshot that summarizes/ supports what Jenne is saying above.
A little over 1 day ago, there was a drop in optical gain. There isn't an indication of a shift in alignment of SR2, or the OMs at that time. AS_C QPD is in loop, so that signal being zero only shows that we weren't using any offsets at the time. The drives on OM1 + OM2 show that their alignment also didn't shift at the time. Since they are used to center the beam on AS_A and AS_B, their drives would need to shift if there is a large shift in the real beam position on AS_C. You can see that for today's lock, where there were large offsets on AS_C, the alignment of SR2 is different, as well as the alignment of OM1 + 2 (indicating that the alignment onto AS_C is different). We might be clipping on the OFI, or elsewhere at the AS port.
Here's an image of the AS camera:
you can toggle through these three in different browser tabs to see that the alignment didn't seem to shift until we added offsets. So all indications are that these offsets aren't good, and we should not be using them, except that they appear to have allowed us to lock the IFO.
Editing to add another screenshot: As Ibrahim checked several times last night, the osems indiczte that SR3 + PR3 haven't moved. In early Tuedsday lock where the optical gain first dropped, the power recycling gain was unchanged, with the introduction of offsets last night it seems slightly lower (and the optical gain also took another step down).This attachment shows that the offsets moved OM3+ OMC, which I did not expect (since the AS centering beams fix the beam axis arriving on OM3, I wouldn't have expected the AS_C offsets to have moved these optics). But they didn't move for the TUesday morning low optical gain lock.
Jennie W, Sheila
Sheila suggested I use the oplevs for SR3, BS, ITMY, ITMX. ETMY, ETMX to compare our alignment out of lock right before we started locking the main IFO (ie. after locking green arms). I chose two times we were in FIND_IR (state 16 in ISC_LOCK).
One of these times was after a lockloss from a very high range lock from Monday afternoon (~157 Mpc).
GPS reference time in lock = 1397864624 (16:43:26 PDT)
GPS reference time in FIND_IR presumably before our current weird alignment/locking problems = 1397868549 (17:48:51 PDT)
The other time was in our last locking sequence Tuesday morning before maintenance Tuesday started. We did not get to NLN but fell out before this at ENGAGE_ASC_FOR_FULL_IFO (state 430 in ISC_LOCK).
GPS reference time in FIND_IR = 1397918790 (07:46:12 PDT)
The main OPLEVS which changed more than 1 micro-radian were ITMY PIT (down by 1.24 microradians)
ETMY PIT (up by 1.02 microradians)
ETMY YAW (down by 1.14 microradians)
ETMX PIT (up by 2.35 microradians)
ETMX YAW (up by 3.21 microradians)
While SRY was locked (last step in initial alignment), we moved the OFI as much as we could, by putting large offsets in the actuator filter banks. No discernable power level changes are visible. This isn't surprising given how weak the OFI actuators are, but seemed worthy of eliminating a possiblity.