IFO is in NLN and in MAINTENANCE as of 18:26 UTC (52 min lock)
With the +Y position detailed in alog 79164, we were able to achieve NLN.
Taking calibration sweeps right now.
We ran the coherence plot (instructions in LowRangeChecks wiki) for 2024/07/16 18:47 UTC with reported 132MPc, plot attached. See lots of coherence with SRCL, PRCL, MICH so we plan to re-tune these feedforwards. The CHARD coherence <15Hz isn't new but the coherence around 30Hz is, it might be coupling through the LSC loops though.
After turning SRCLFF gain down from 1.14 to 1.0, the coherence looked a lot better and the range increased to 145MPc.
We took FF measurements for SRCL, MICH and PRCL, saved to /opt/rtcds/userapps/release/lsc/h1/scripts/feedforward. Before fitting these we found we didn't like this IFO alignment (79172).
As looked at in 79117 and 78399. the OFI's temperature control changes as we power up.
In a nominal lock before this alignment change, VOLTAGEBACK changes 0.0091V and the temperature wobbles 0.0031 degC: plot.
In today's power up at the new alignment, VOLTAGEBACK changes 0.0267V and the temperature wobbles 0.0135 degC: plot. This is 3 to 4.5 times larger than usual and larger than the changes we saw May 28th/29th 78442 power-ups: 0.0458V and 0.0097degC plot.
Note that we don't understand if this is damaging at all but it makes us think that we are closer to clipping the OFI or have the beam spot in a different position in relation to the TEC/thermistor: TEC is on bottom of the OFI TGG holder (OMC side): G1902217 slide 22. Unsure location of thermistor.
CS Mains mon trend attached.
At the beheast of Jeff, I have revived this script from 2011 written in python2 to generate medm matrixes. Updating seemed pretty straightfoward, I ran a builtin automated conversion to start, "2to3 file.py" (-w if you want it to actually write the changes).
I tested it by running "python3 generate_KisselButton2024.py 10 4 H1:OMC-ADC_INMTRX" and copy pasting the output into a blank .adl which gives me this 10x4 matrix button with a connected display window based on the input channel arg. It needs a minumum of 3 args: #rows #col channel. You can also specify the size of the button/matrix, and the linked related display .adl file(you can put the whole path).
The new script is found at /opt/rtcds/userapps/release/cds/common/scripts/generate_KisselButton2024.py, and is svned. I left the old script where it is (the same name minus the 2024).
Amazing -- thanks Ryan!
AS_A Counts after shuttering yield a ratio of 325 ppm. Find screenshot below. Template for this ndscope is saved under:
/opt/rtcds/userapps/release/sys/h1/templates/ndscope/fastshuttercheck.yaml
This is with the +Y alignment spot in SR3 detailed in alog 79103:
Start | +P move | -P move | +Y move | -Y move | |
SR3 P slider | 438.7 | 651.1 | 138.7 | 438.7 | 438.7 |
SR3 Y slider | 122.2 | 122.2 | 122.2 | 322.2 | -167.8 |
Tue Jul 16 08:07:49 2024 INFO: Fill completed in 7min 45secs
Jordan confirmed a good fill curbside.
Workstations were updated and rebooted. This was an update to OS packages. Conda packages were not updated.
TITLE: 07/16 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 1mph Gusts, 0mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.05 μm/s
QUICK SUMMARY:
IFO is attempting to lock based on an SR3 +Y position that Sheila believes may have promise. See alog 79103.
Tuesday Maintenance has been deferred for this week as we work on high priority IFO Locking Issues.
Other:
H1:PEM-CS_DUST_PSL102 WARNING: dust counts did not change, please investigate
In initial alignment we trigger the length feedback to SRM in SRY. In the last day I've been setting these thresholds to zero so that we aren't triggering, and this has worked fine many times now. However, when the alignment is poor and the build ups are too low for the triggers we have been using, the WFS don't work well. I suspect that we've been using the LSC triggers to make sure the alignment is good enough for the WFS to work.
We could consider a dither alignment instead of WFS, since the length lock seems very robust. Or, as a first step let's move the threshold to a check that's run before ASC comes on, rather than preventing length locking of the cavity.
Chris is actively working on refitting our VEA HEPA filters. This required swapping of the fans at the both EX and EY. The swap occurred between 7:10a-7:15a PST. C. Soike T. Guidry
TITLE: 07/16 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY: I've put the detector in IDLE for the rest of the night. Sheila and I found that the +Y alignment (from 79103) doesn't seem to be the best (see Sheila's thoughts on this alignment: 79155), and so we decided to try out the +P alignment. We were eventually able to get DRMI to lock with this, but the beamspot on AS_AIR didn't look great(attachment1), and once ASC kicked in we started getting mode hopping(attachment2), with DRMI staying locked. We got through to CHECK_AS_SHUTTERS and ran the fast shutter test and got values for ASC_AS_A_DC_NSUM_OUT_DQ of 3.2k before the shutter closed, and ~1 when the shutter closed. giving 250ppm (attachment6). Unfortuately, we lost lock from CHECK_AS_SHUTTERS a while after and I haven't been able to get DRMI locked since then, but PRMI locks quickly.
LOG:
IFO in +Y alignment (79103)
23:00 Troubleshooting and at ENGAGE_ASC_FOR_FULL_IFO
23:07 Lockloss from DARM_OFFSET
23:21 Lockloss FROM CARM_TO_TR
23:49 Got to DARM_OFFSET (offset at 9e-5)
- 23:50 Dip in power recycling gain started, Ryan S changed darm offset to 6e-5 and was able to save lock, we continued to sit in DARM_OFFSET
23:58 I took ifo to DOWN and into SR2_ALIGN in ALIGN_IFO guardian
- Comparing AS_{A,B}_NSUM_OUT_DQ and AS_C_NSUM_OUT_DQ values to those channels' values from second to last initial alignment before last NLN on Friday morning and trying to rase those channels' values to match previous alignment (attachment3). horizontal cursors are placed at the bounds where these channels were during the second to last lock's IA.
04:12 Back to DOWN and tried to get DRMI locked
- locked PRMI, but doesn't look great after ASC (attachment4)
- DRMI wouldn't lock for more than a second or two, and it looks Bad (attachment5).
04:38 Went to DOWN, decided to try +P location
- Went into MANUAL_INITIAL_ALIGNMENT, went to SR2_ALIGN, changed power to 10W, and moved SR2 and SR3 to the locations given in 79135. Then Sheila moved SRM to line up the beams on AS_AIR. We went through SRC alignment and offloaded
IFO in +P alignment (79103)
05:24 To DOWN, started relocking
05:39 DRMI locked itself after 10 minutes of trying
- Looked weird (attachment1)
- Lots of SRM saturations while in OFFLOAD_DRMI_ASC
- After ASC, DRMI started splitting while staying locked (mode hopping) (attachment2)
05:42 CHECK_AS_SHUTTERS ran fast shutter test (attachment6)
- Value in AS_A before shutter close = 3.2k and after = ~ 1.0 counts; 250 ppm overall for ASC_A - not bad, seems like beam spot is fully blocked
- Compare to Keita's earlier tests 79141, 79147
06:04 Lockloss from CHECK_AS_SHUTTERS, trying to relock
- Cannot seem to get DRMI to lock again, but PRMI will lock quickly
- Cycled thorugh PRMI and MICH and still couldn't get DRMI
07:44 Put detector in IDLE for the night
Oli was searching for a better throughput to AS diodes when I logged in, based on the observation earlier that in the +Y SR3 position from 79103. As I commented 79153 I think that is explained by poor DRMI build ups.
Because we did have a CARM offset reduction lockloss earlier from the +Y position, I thought we should try a different one, and we opted for -P from 79103. The attached screenshots show the AS camera and slider settings after running SRC alignment, and again after locking PRMI. The AS camera looks strange in PRMI. The third screenshot shows the single bounce beam with 10W (after running full initial alignment), it looks clipped on the AS camera. Our conclusion is that there is some clipping or other problem in the -P position, and it isn't a good place to try locking.
Oli has now moved up to the position called positive pitch in 79103, screenshot after running SRC align is attached.
A PCAL EndX Station Measurement was done on 2024-07-02, the PCAL team (Dripta B. & Francisco Llamas.) went to EndX with Working Standard Hanford aka WSH(PS4) and took End station measurements using T1500062-V17 as a guide.
Measurement Log
First thing we did was take a picture of the beam spot before anything is touched!
Martel:
Martel Voltage source applies voltage into the PCAL Chassis's Input 1 channel. We record the GPStimes that a -4.000V, -2.000V and a 0.000V voltage was applied to the Channel. This can be seen in Martel_Voltage_Test.png . We also did a measurement of the Martel's voltages in the PCAL lab to calculate the ADC conversion factor, which is included on the above document.
Plots while the Working Standard(PS4) is in the Transmitter Module during Inner beam being blocked, then the outer beam being block, followed by the background measurment: WS_at_TX.png.
The Inner, outer, and background measurement while WS in the Receiver Module: WS_at_RX.png.
Plots of inner, outer, and background measurement while WS sphere is in the RX enclosure with both beams on it and RX with both beams : WS_at_RX_BOTH_BEAMS.png
Then placed the Working Standard (PS4) in the path of the INNER Beam at the TX module.
Then the Working Standard (PS4) in the path of the OUTER Beam at the TX module.
A background measurement.
Took the Working Standard and put it in the RX module to get the INNER Beam.
Then the OUTTER Beam in the RX Module.
And a Background.
We remove the beam block and give the Working Standard Both Inner and Outer Beams at the SAME TIME while it's at the RX module.
We also put the RX sphere back to the RX module and put both beams on it at the same time. Like nominal opperation when the PCAL lines are turned off.
Then we take a background.
The last picture is of the Beam spots after we had finished the measurement.
All of this data and Analysis has been commited to the SVN or GIT:
https://git.ligo.org/Calibration/pcal/-/tree/development/O4/measurements/LHO_EndX/tD20240702?ref_type=heads
This adventure has been brought to you by Dripta B. & Francisco.
This is a late alog due to the updates in the procedure and the trends document.
Today we conducted a brief test on the SUS-EX ESD DAC vs the SUS-EX LD32 DAC, both driving the PEM AA chassis at end X per WP11976. This testing was cut short due to relocking attempts. We changed the cable to the PEM output ~1:10pm PST, and replaced the cable back to its original position at ~2:15pm PST. We are still looking at the results of this testing.
F. Clara, R. McCarthy, M. Pirello, D. Sigg
We compared the output of the 20-bit DAC with the LIGO DAC. There is delay of about 90us. More than we would have expected. Decimation filter delay for a ~30kHz cut-off is about 14us. The filters are getting processed and sent to the DAC at a rate of 1MHz. So, maybe ~1-3us processing delay.
Sheila, Keita, Ryan Short, Camilla and TJ
Part way through initial alignment, we stopped and moved SR3 towards the positive yaw spot from 79103, and used the SR2 osem values from that time. A small manul adjustment of AS_C was needed, otherwise initial alingment was uneventful.
With DRMI locked, we ran the fast shutter test that Keita and Ryan S have both looked at.
We also looked at the ratios of AS PDs to compare to 78667, they are most similar to the good times in that alog. After these checks we have decided to try locking.
Fast shutter behavior is attached. It's working fine, but the throughput to HAM6 is ~14% down compared with before.
Before the shutter was closed, ASC-AS_A_DC_NSUM was ~3.7k counts, and ~0.75 counts after (fractional number because of decimation to 2k). That's about 200ppm.
However, it used to be ~4.3k and 1cts on one of the "happy" plots alog 79131, 3.7k/4.3k~0.86, so the throughput to HAM6 seems to be ~14% lower than before.
IFO lost lock even before going DC and we had another FS test forced by the guardian, and it didn't fail, but the throughput is even worse.
A_DC_SUM=3.4k when the shutter was open, Closed/Open ratio is about 1000ppm, and a tiny part of the beam is being missed by the shutter (attached). Note that I'm NOT eyeballing the "open" Y cursor in log scale, I set it while in linear Y scale but changed it to log to show that the power after the shutter was closed seems to be measurably larger than it should be.
Maybe this is going on because of tiny alignment differences from lock to lock, but anyway this doesn't look to be the place we want.
I think the reason that the power levels were low on AS_A and AS_B in the DRMI lock mentioned above was because DRMI was poorly aligned, not due to excess output arm losses.
Last good lock: DRMI locked, POP18 NORM was 65.3, POP90 NORM was 9.7, AS_A DC 4173 cnts, AS B DC was 4395 and AS_C was 0.0517 (Watts into HAM6).
In today's earlier DRMI lock, POP18 NORM was 55.35 (15% lower than the good lock) POP90 NORM was 12.27 counts (30% higher). I think this indicates that DRMI was poorly aligned in this earlier lock, not necessarily that it was a bad spot through the OFI.
To help characterize the recent issues we've seen with IFO alignment that are familiar to the output change in late April, I moved SR3 in 10W single-bounce configuration with the SRC2 ASC loop on so that SR2 would follow (ALIGN_IFO's SR2_ALIGN state), much like in alog77694. I moved SR3 far enough in each direction (+/- pitch and yaw) to bring the ASC-AS_C_NSUM signal back up to our target value of 0.022, which I was successful doing in every direction except -Y, where I could only bring AS_C up to 0.018. In the three directions where I was successful, the beam spot on the AS AIR camera looked much more like the clear circle we're used to seeing and less like the upside-down apostrophe we have now.
It seems that our old output spot (starting place in TJ's alog) is still bad (-Y direction with SR3 from current place) since that was the only direction where I couldn't get the AS_C power back up.
Slider values of SR3 after each move:
Start | +P move | -P move | +Y move | -Y move | |
SR3 P slider | 438.7 | 651.1 | 138.7 | 438.7 | 438.7 |
SR3 Y slider | 122.2 | 122.2 | 122.2 | 322.2 | -167.8 |
Attachment 1 is the starting AS AIR camera image (and our current spot), attachment 2 is after the +P move, attachment 3 is after the -P move, and attachment 4 is after the +Y move.
|
Before |
+P move
|
-P move
|
+Y move
|
-Y move
|
Time
|
1404952764
2024/07/14 00:39:06 UTC
|
1404955465
2024/07/14 01:24:07 UTC
|
1404958133
2024/07/14 02:08:35 UTC
|
1404959716
2024/07/14 02:34:58 UTC
|
1404963518
2024/07/14 03:38:20 UTC
|
H1:SUS-SR3_M1_OPTICALIGN_P_OFFSET
|
438.7 |
651.1
|
138.7
|
438.7
|
438.7
|
H1:SUS-SR3_M1_OPTICALIGN_Y_OFFSET
|
438.7 |
122.2
|
122.2
|
322.2
|
-167.8
|
H1:SUS-SRM_M1_DAMP_P_INMON
|
-1033.7 |
-1035.2
|
-1036.3
|
-1036.1
|
-1037.1
|
H1:SUS-SRM_M1_DAMP_Y_INMON
|
913.7 |
914.0
|
914.1
|
914.1
|
914.2
|
H1:SUS-SR2_M1_DAMP_P_INMON
|
597.7 |
-871.6
|
2660.2
|
614.7
|
566.2
|
H1:SUS-SR2_M1_DAMP_Y_INMON
|
1125.3 |
1179.9
|
1069.3
|
1878.8
|
-72.4
|
H1:SUS-SR3_M1_DAMP_P_INMON
|
-290.2 |
-57.7
|
-619.4
|
-297.9
|
-279.1
|
H1:SUS-SR3_M1_DAMP_Y_INMON
|
-411.0 |
-425.2
|
-390.4
|
-256.9
|
-633.7
|
Here are the OSEM values, so that Alena can cotinine her 78268 analysis of the OFI beam spot position.
Adding a screenshot of sliders for the -P alignment above, after initial alignment.
Yesterday, the fast shutter test failed due to dark offsets in the AS WFS DC NSUM channels.
The guardian started running the TwinCAT testing code, which works as intended: it sends a close command to the trigger logic, which in turn fires the fast shutter. The fast shutter works fine as can be seen on the HAM6 geophones. The slow controls readbacks also indicate that both fast shutter and PZT shutter are closed no later than 200ms after the trigger. However 0.5sec after the guardians started the test, it checks the AS WFS DC NSUM outputs and compares them against dark offset limits of ±15. Since the dark offset on WFS B was over 30, the guardian then sent an abort command to the TwinCAT code and reported a failure.
That story agrees with my observation on Friday night when I started looking at the FS after 11:15 PM.
Sheila reported (79092) that the Fast Shutter reopened only after ~50ms or so. It seems that the low voltage drive to keep the shutter closed was not working. 1st attachment shows the last time that happened at around 23:10 local time. Daniel points out that the shutter was in an error state at that time but that was after Ryan power cycled the FS driver. We don't know exactly what kind of state the fast shutter was in here.
The next time the HV firing was tested was at 23:23 local time (2nd attachment), the shutter was kept shut (i.e. low voltage thing was somehow working) but there are two things to note.
The last FS test I've done was 00:12 local time on Jul/13 when the error was cleared, with the smaller power than nominal (3rd attachment). Bouncing was as bad but the power coming to HAM6 was smaller (see the trigger power at the top left). AS_B_NSUM was somewhat smaller (more like 10).
The reason why AS_B_NSUM is worse is because I reduced the analog DC gain by a factor of 10 and compensated for that by digital gain. The effect of analog offset as well as ADC/electronics noise are 10x worse than AS_A. I adjusted the dark offset while IMC was unlocked but we can probably increase the threshold to 30 or so if it continues to bother us.
Bouncing behavior might be more serious as it could mean that the beam was close to the end of the travel of the FS mirror (and it was bad on Friday because of funny alignment), or low voltage drive was somehow still funny. I suspect the former.
Here is my attempt at recreating the (rough) timeline of events related to the fast shutter Friday night (all times in UTC):
Seems like ASC-AS_B_DC gain was probably a red herring, important thing is that the beam got uglier/bigger at some point, therefore a part of the beam was not blocked by the fast shutter.
First attachment shows when ASC-AS_B_DC gain switch was flipped from x10 to x1 on Tuesday. You can see that the Fast Shutter has been firing OK until Friday evening.
The rest of the plots show the FAST Shutter test done by the Beckhoff at three different points in time, i.e. the last test before my AS_B_DC change ("happy" Monday July/08 ~16:37 UTC or 9:37 local), the first one after my AS_B_DC change ("happy" July/11 4:43 UTC or Wed July/10 21:43 local time), and the first time the FAST shutter went into the error mode ("sad" Jul/13 4:56 UTC or Fri Jul/12 21:56 local time). The last one is when Sheila and Ryan started having problem.
Important points are:
From these, my conclusion is that the beam position on the fast shutter mirror was pretty much the same in all three tests, but the beam was pretty ugly for the "sad" plot as was witnessed by many of us on AS port camera. Because of this a part of the lobe was missed by the Fast Shutter. Centering an ugly beam on AS_C might have complicated the matter.
Later, when I forced the test with much lower power, the error was cleared because even though the ugly beam was still there the power went lower than the "shutter closed" threshold of the guardian.
I don't fully understand who did what when during the time the shutter was in the error state (it includes people pressing buttons and power cycling the driver and then pressing buttons again, and I certainly pressed buttons too).
Looking at this, and since Daniel agrees that the Fast Shutter has been working fine, my only concerns about locking the IFO are:
FYI, fast shutter has never failed to fire up and never failed to keep shutter closed when the trigger voltage crossed the threshold of 2V since I connected the ASC-AS_C analog sum to the PEM ADC to monitor the trigger voltage at 2kHz at around Jul/11 2024 18:30 UTC, i.e. in the past 5 days.
In other words, FS has never failed when we actually needed it to protect AS port.
In the 1st attachment, top is the trigger voltage monitored by the ADC. Red circles indicate the time when the trigger crossed the threshold. We've had 12 such events, including 4 that happened after Friday midnight local time.
Middle plot shows when the fast shutter was driven with high voltage (sensed by GS13). Bottom shows the AS_A_DC_NSUM and B_DC_NSUM.
The reason why it looks as if the trigger level changed before and after t=0 on the plot is because a 1:11 resistive divider with a total resistance of 10kOhm was installed at t=0. Before that, the 2V threshold was 32k counts, 2980 counts after.
The rest of plots show that each and every time the shutter was triggered by the trigger voltage rather than a test, it always fired up and kept the shutter closed.