Keita, Sheila,Robert, TJ, Jennie, Corey, Tony
Dave and Gerardo alerted us that there have been vacuum spikes after locklosses in HAM6 yesterday. We are looking at what circulating powers were and what the lockloss transient was at the time of locklosses that caused and didn't cause a vacuum spike in HAM6. Here is a comparison table:
time | AS_C peak | POP A LF | which alignment | spike? |
6/7 1:12:35 | 0.938*10 = 9.378 | 5505 | bad | 3.6e-8 |
6/7 3:04:08 | 1.35*60 = 81 | 26739 | bad | 4.28e-8 |
6/7 03:51:29 | 1.05*60 = 63 | 27032 | bad | 1.5e-7 |
6/7 23:15:54 UTC | 0.342*2 =0.684 | 1109 | good | no |
6/7 20:54 UTC | 1.58*2= 3.16 | 651 | good, PRG was tanking | no |
In the bad alignment, the beam was clipped on it's way to AS_C, so that diode wasn't catching all the power it would have in the good alignment. AS_A and AS_B saturate at the time of locklosses.
The spike in vacuum pressure was higher with each successive lockloss, although the power on AS_C was largest in the second one.
TITLE: 06/07 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 5mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.14 μm/s
QUICK SUMMARY:
IFO has been seeing vacuum pressure spikes during locklosses potentially coming from HAM6.
Currently Holding in OFFLOAD_DRMI_ASC. and will be touching up alignment by hand.
A plan is being made to determine the corrective action for the Pressure spike issue which MIGHT be due to a beam heating up something that should not be heated up.
Lockloss, getting bck to OFFLOAD _DRMI_ASC to manually align.
TITLE: 06/07 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Tony
SHIFT SUMMARY:
Continuing OFI pointing work: Undid the pointing from last night and returned to a different pointing.
Other big item was toward the end of the shift when Dave & Gerardo contacted the Control Room regarding pressure glitches seen last night which corresponded with (3) higher power locklosses last night. (We are no longer at this pointing as noted above though, but this took all the focus of the last hour or so.)
LOG:
Sheila D, Jennie W, Corey G, TJ S
Yesterday we tried a new SR2&3 alignment combination for hopes of better OFI throughput, but after a hrad fought alignment battle, this alignment proved to not work out (alog78290). This morning we started the process to move back to our SR positions that we have been running for the last month or so.
The process was the same that we have been running: walk SR2 and SR3 in a single dof while trying to not fall off AS_C, AS_B, or AS_A, then correct with picos when unable to stay on the QPDs. Repeat with the another dof. This got us back to our target SR positions, but OM3 and OMC were either saturating or getting close to it. Based on part of Keita's recommendation for how to do this move (alog78297), we moved the OMs and some picos enough to engage the OMC ASC, which controls OM1&3 to steer into the OMC QPDs, then moved OM2 to relieve some of the strain on OM3&C. This required a DOF2TT matrix change as in Keita's alog. Eventually we were able to get all QPDS (AS_{A,B,C} & OMC_{A,B}) to be centered with fairly low drive output on the suspensions.
Corey ran a manual initial alignment and brought SRM back to near where it has been. When relocking we first ran into the DARM offset issue that we have been seeing lately (see alog78273), and I didn't catch it in time. The next time around Sheila put the SCR1 Y offset of in and changed the DARM offset and that was enough for us to lock the OMC and move onto DC readout. We've now heard of some other issues, so we are holding here and not advancing over 2W yet.
J. Kissel, Yesterday, I built another DB9 "shorting plug," similar to that from LHO:67465 that shorts all DB9 pins to Pin 5 without shorting to the backshell of the DB9 connector -- see pictures in the attached 2024-06-06_SEI-C2_DifferentialChanShortingPlug.pdf. Yes, the internal wiring looks sloppy, but I've confirmed that it functions as desired electrically. While the IFO was recovering from a routine lock loss, I installed the plug into the open-and-unused "In 25-28" port of the aLIGO HAM-ISI Anti-Aliasing Chassis (D1000269) in U28 of SEI-C2 in the Corner Station CER. See pictures of where in the attached 2024-06-06_SEI-C2_ShortingPlugInstalled.pdf. These 4 channels are otherise unused channels on the seih23 IO chassis' ADC2, which otherwise houses the HAM3-ISI system's sensors and coil driver readbacks (channels ADC2_24 thru ADC2_27). These channel inputs were originally intended to be for analog copies of the ground STS2 (which are now all T240s), created because in the early 2010s, we didn't yet trust the inter process communication (IPC) system. We don't do this, and honestly never ran the cabling -- we've been successful in mapping all ground analog signals from GND T240s into the seib1 IO chassis, reading them out with h1isiitmy front-end model, and then shipping them to h1seiproc via digital IPC, processing them into a sensor correction signal, and then sending over IPC again to each ISI, including HAM3. Note -- the HAM ISI wiring diagram, D1101576, inaccurately portrays what channels are unused on this AA chassis at LHO, so I also took the time to create some "as built" photos of the SEI-C2 rack, which are are now linked from the rack's S-number DCC page, S1301863 and will reach out to Dean/Arnaud to get it updated. I do so for two reasons: - For the future HAM3-HAM2 SPI pathfinder system (E2400121), the pitch / yaw "two-way optical lever" QPD system will be limited by ADC noise, so I want to quantify that noise in the real LIGO IFO system down to as low a frequency as possible. Indeed, this rack, and this ADC is what may be used for the pathfinder -- during the conceptual design review, a suggestion was made to look for spare channels in the SEI system before assuming the SPI needed to buy another ADC card; see M2300216. - Given recent DuoTone investigations (LHO:78238 and references therein), these channels might prove as an interesting base-line for "normal" shorted ADC channels, like the shorted channels on the low-noise, 524 kHz ADC from LHO:67465. The read back channels for these will be H1:IOP-SEI_H23_MADC2_TP_CH24 H1:IOP-SEI_H23_MADC2_TP_CH25 H1:IOP-SEI_H23_MADC2_TP_CH26 H1:IOP-SEI_H23_MADC2_TP_CH27 They've always existed, it's just that the differential inputs are now connected to the well-defined 0V GND reference of the AA chassis (which is connected to the same 0V GND reference for the ADC) -- rather than floating. They're sampled at 65536 Hz and not stored in the frames, so only live studies will be possible. Stay tuned for measurements.
Fri Jun 07 10:09:48 2024 INFO: Fill completed in 9min 44secs
Travis confimed a good fill curbside.
For FAMIS 26249:
Laser Status:
NPRO output power is 1.81W (nominal ~2W)
AMP1 output power is 66.81W (nominal ~70W)
AMP2 output power is 137.4W (nominal 135-140W)
NPRO watchdog is GREEN
AMP1 watchdog is GREEN
AMP2 watchdog is GREEN
PDWD watchdog is GREEN
PMC:
It has been locked 9 days, 20 hr 40 minutes
Reflected power = 20.94W
Transmitted power = 106.2W
PowerSum = 127.2W
FSS:
It has been locked for 0 days 10 hr and 48 min
TPD[V] = 0.8227V
ISS:
The diffracted power is around 2.5%
Last saturation event was 0 days 10 hours and 48 minutes ago
Possible Issues:
PMC reflected power is high
TITLE: 06/07 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: n/a
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 11mph Gusts, 8mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.17 μm/s
QUICK SUMMARY:
H1's been down for roughly the last 10hrs and recovery/troubleshooting issues with H1 continue with Observatory Mode set to COMMISSIONING. There were dust alarms in the optics lab this morning for a few minutes. Winds are low and microseism has slowly continued a downward trend to the 50th percentile.
Addendum: After quickly chatting with Sheila, the plan is to do single-bounce configuration work this morning (she is getting set-up now).. This is to undo the new OFI (Output Faraday Isolator) spot from yesterday (which had issues due to limits in the OMC suspension). Also trying to recover from Maintenance Day.
Jenne, TJ, Sheila, Ibrahim
TITLE: 06/07 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: TJ (in name only)
SHIFT SUMMARY:
IFO is in DOWN and staying that way until tomorrow.
The remainder of the H1 EVE Shift and the entirety of the upcoming OWL Shift have been cancelled due to Locking issues with the new OFI configuration.
We have not been able to lock since yesterday's short 2.5 hr lock, ending at roughly 10:30 UTC.
Comissioners were tirelessly testing out a new OFI spot configuration when my shift started and despite their determination, it does not seem to be in a good spot. Jenne and I confirmed that IFO_OUT suspensions do not saturate during powerup but it seems that they tend to do so (only OM2 and OMC) at LASER_NOISE_SUPPRESSION. Thus, we have elected to revert the changes tomorrow morning. Check out alog 78290 for more information on this comissioning work.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
21:03 | SAF | HAZARD | LVEA | YES | LVEA is Laser HAZARD | 23:32 |
21:26 | SQZ | Terry, Kar Meng | Opt Lab | Local | SHG work | 23:56 |
23:12 | CAL | Francisco | PCal Lab | N | Measurements | 23:31 |
00:08 | PEM | Robert | LVEA | Y | Turning off amps and taking pictures | 00:25 |
Sheila, Francisco, Louis
We would like to measure the sensing function when the IFO is thermalized, with different DHARD ASC loop gains, to see if we still have the ASC cross coupling impacting the sensing function as we saw back in O3: 50511
Based on the measurement in 77857 we thought we could increase the gain of DHARD P from -30 to -45. This worked fine, so we also tried increasing the gain of DHARD Y from -40 to -60, we were also stable with both of these gains increased by 50%. I then increased DHARD Y further stepping up to -80 where we lost lock.
Gabriele, Sheila
Between the locks on June 3rd and June 4th, our LSC feedforward became mistuned (more dramatically than before). This might explain most of our range drop this week. We don't know why this happened, there aren't any known changes that occured, other than a stretch of windy days with poor locking.
This morning we were briefly locked in our "post April 24th alignment", and Gabriele walked me though doing the iterative feedforward measurements.
I've updated the README.txt in userapps/lsc/h1/scripts/feedforward with instructions for the iterative measurement:
go to NLN_CAL_MEAS
To turn off FF: SRCLFF1 (or MICHFF) to DARM: turn SRCLFF1 (or MICHFF) gain to zero, turn off the FF filter but leave the high pass on, turn off the input to SRCLFF1, set gain to 1. Run excitation with the SRCLFF template
SRCL to DARM: nominal configuration with MICH and SRCL FF on, in NLS_CAL_MEAS. Run the template for SRCL excitation
When you measure MICH to DARM you leave everything in nominal condition
When you measure SRCL to DARM you leave everything in nominal condition
When you measure SRCLFF to DARM you leave MICHFF in nominal condition, turn off SRCLFF as aboves
When you measure MICHFF to DARM you leave SRCLFF in nominal condition, turn off MICHFF as above
The measurements are checked in at userapps/lsc/h1/scripts/feedforward
SRCLFF_excitation_ETMYpum_2024_06_06.xml
NoLP_MICHFF_excitation_ETMYpum_June62024.xml
MICH_excitation_June62024.xml
SRCL_excitation_June62024.xml
Here are two fits that should improve MICH and SRCL FF
SRCL
zpk([-7.169784531472987+i*46.80222713776738;-7.169784531472987-i*46.80222713776738;-50.91705199165386+i*64.16016193457278;-50.91705199165386-i*64.16016193457278;-42.15501976438775-i*1098.232422972419;-42.15501976438775+i*1098.232422972419;-145.0093557348077-i*1089.461704122895;-145.0093557348077+i*1089.461704122895;-245.0829862689482+i*1107.79921936364;-245.0829862689482-i*1107.79921936364;-141.5739067943562;-144.9209247430132;-826.3641092498208;-852.1997030446448],[-2.881922149238943+i*32.00142415651089;-2.881922149238943-i*32.00142415651089;-39.6556740022373+i*916.0295759541193;-39.6556740022373-i*916.0295759541193;-230.3431403668346+i*2309.933027270674;-230.3431403668346-i*2309.933027270674;-39.76964022276446;-39.8605595541308;-39.89837525205524;-39.91157191605507;-40.46668526791298;-2581.144039450426;-2671.009338936645;-2738.495590062898],0.4767541992000439)
MICH
zpk([-0.1470884781758559+i*48.42475784167187;-0.1470884781758559-i*48.42475784167187;-24.82947443442181+i*50.41982768133377;-24.82947443442181-i*50.41982768133377;-1.195340221327271+i*60.10254083145923;-1.195340221327271-i*60.10254083145923;-0.8119687656946064+i*112.301511110354;-0.8119687656946064-i*112.301511110354;-94.78274149473425+i*118.9486474187639;-94.78274149473425-i*118.9486474187639;-307.5544495976067+i*199.6473319209523;-307.5544495976067-i*199.6473319209523;-365.1498461668634+i*797.5973180592285;-365.1498461668634-i*797.5973180592285;-264.514660945466+i*941.2100034065085;-264.514660945466-i*941.2100034065085;49.35448274331864;-185.3386536745414],[-1.619706160754901+i*48.82075959570737;-1.619706160754901-i*48.82075959570737;-1.342047624329135+i*59.97321034526349;-1.342047624329135-i*59.97321034526349;-0.7898795886248342+i*112.7249598047579;-0.7898795886248342-i*112.7249598047579;-310.8951954733114+i*592.1396997441478;-310.8951954733114-i*592.1396997441478;-167.320180125188+i*927.5461319412049;-167.320180125188-i*927.5461319412049;-33.94826038138513;-33.98691038181345;-34.44985837050922;-35.1231562638948;-189.8775386189932;-309.2218677305598;-375.827056617601;-1502.151637570457],-8.520785430534616)
Jenne, Ibrahim
Trended the net movements of some OM suspensions during powerup from our most recent lock acquisition (2024-06-06 10:21UTC to 2024-06-06 15:56 UTC). This was done during today's comissioning to know whether our new OFI alignment configuration would saturate our IFO Out suspensions during powerup. Therefore, this data is just one reference of how much these suspensions usually move during powerup. For details on the actual OFI move today, refer to alog 78290. I've attached an identical (and prettier) sheets screenshot of the table below.
Suspension |
Net Powerup Delta |
Net Towards 0? |
Local Extrema Delta |
Local Extrema Towards 0? |
Remarks |
OMC (131k) |
|
|
|
|
OMC Results are to the nearest thousand |
T1 |
0 | N/A | 0 | N/A | |
T2 |
29000 | Yes | 13000 | No | |
T3 |
24000 | Yes | 14000 | No | |
LF |
9000 | Yes | 7000 | No | Net is largely unchanged |
RT |
13000 | Yes | 7000 | No | |
SD |
0 | N/A | 0 | N/A | |
OM2 (32k) |
|
|
|
|
OM2 Results are to nearest 100 |
UL |
100 | No | 900 | No | |
LL |
1100 | No | 1900 | No | |
UR |
1200 | No | 1900 | No | |
LR |
600 | No | 800 | No | |
OM3 (32k) |
|
|
|
|
OM3 Results are to nearest 100 |
UL |
1200 | Yes | 1100 | No | Amplitude of fluctuation dips below and above baseline (so while extrema trends away from 0 there are about 200 cts towards 0 with each swing |
LL |
1100 | No | 700 | Yes | Amplitude of fluctuation dips below and above the baseline (so while extrema trends towards 0, there are about 200 cts away from 0 with each swing) |
UR |
700 | No | 600 | Yes | Amplitude of fluctuation dips below and above the baseline (so while extrema trends towards 0, there are about 200 cts away from 0 with each swing) |
LR |
1300 | Yes | 1000 | No | Amplitude of fluctuation dips below and above baseline (so while extrema trends away from 0 there are about 200 cts towards 0 with each swing |
Sheila, Jenne, TJ, Ryan, Keita, Jennie W
Today we had trouble locking from the start of the commissioning period. This seems to have been a problem with OMC locking - see Ryan's log for further details.
We scanned the OMC in single bounce anmd everything seemed to look ok.
We managed to power up but then eventually lost lock after Sheila measured LSC FF and Robert took some photos at viewports.
Alena had suggested a better spot we could be aligned to on the OFI (second column on second last page), so since we have been having locking problems since maintenance on Tuesday we decided to unlock and try this move.
The best spot suggested is so far off that we know we will have to use the HAM6 picomotors to recentre the beam on the AS_A, AS_B and AS_C QPDs.
TJ and Jenne slowly brought us to this new spot (between two t cursors in this image) while also using picomotors 2X, 3X, 3Y, 1X to keep us centred on AS_A, AS_B and AS_C. During this time we were also trying to run the DC centering loops on OM1 and 2 to keep us centred on AS_A and AS_B, unfortunately we then ended up in a situation where we were saturating OM1, 2 and 3 and OMC suspensions at different points. Turning off the DC centering loops relieved OM1 and 2.
We couldn't get past PREP_DC_READOUT_TRANSITION.
Before doing this, at 22:50 UTC Sheila and I measured the spot positions on the SRM and SR2 mirrors using the A2L gains.
Sheila tried to releive the OMC and OM3 suspensions using their sliders but eventually we realised that we might have to do a combo of this with more pico-motoring to lock the OMC and power up.
This is shown in this image, however, these efforts eventually unlocked us.
Here is another ndscope showing the full days efforts with the ISC_LOCK guardian state also.
If we decide to stick with this new alignment, we'll need to update the thresholds in SRC align when acquiring SRY so that H1:LSC-SRCL_TRIG_THRESH_ON is something like 0.003 (so, half of its current value of 0.006), and the _OFF is similarly lowered. Maybe not a full half, but some lowering.
Also, we are somewhat concerned about the throughput in this position. AS_C yesterday when centered had an NSUM of about 0.0225, but today when centered is about 0.18. However, when we were doing SR2 single-bounce alignment as part of initial alignment, AS_C was at 0.0218. So, some of the loss of NSUM on AS_C happens when we center on it. This is rather consistent with AS_A and AS_B sums thinking that we still have similar throughput through the OFI today as compared to yesterday.
Also, when we were at PREP_DC_READOUT_TRANSITION, I spent a while with Ibrahim offloading OM3 and OMC by changing the spot position (via picomotor) on the AS WFS. I think Jennie shows this in her second to last plot.
I spent a while moving things to get the OMC QPDs closer to their setpoints. Then, with the OMC ASC loops engaged (with OM3 and OMC not quite saturating), I moved the picomotors in front of AS_A and AS_B to get OM3 and OMC farther from saturation. This part (the offloading, after OMC ASC was on) is shown in the attached image. After I had offloaded OM3 and OMC, I requested OMC_LOCK to lock the OMC, and we were able to go to DC Readout. As Jennie notes, after we got to 10W I was worried that OM2 was going to saturate, so I made one more click on Motor 2 to change the pointing on AS_B, and that caused a lockloss (even though the same clicks at 2W on RF DARM were safe). The last little bumps in the most recent 5 mins of this xaxis are the power up to 10W.
When you make a big alignment change upstream of HAM6 and you find that WFS centering won't work without pico, the only cure is pico-ing, but confirm that the OMC QPDs are OK before locking IFO.
The right sequence is something like this:
Using single bounce beam, enable WFS centering. If nothing rails, turn on the OMC ASC, nothing should rail again, be happy.
If OM1 and/or OM2 rail by WFS centering, eventually you'll have to pico, but don't pico right away. Instead, do the following.
This is pretty much the same as what you do after changing AS path and/or OMC in HAM6. By working on AS path alignment before locking IFO, you don't have to worry about IFO lockloss by upsetting ASC-AS_A, kicking HAM6 electronics by HV pulses sent to picos and/or making huge vibration in HAM6.
When manually aligning the beam into OMC using OM1 and OM3 (or OM1 and OM2, or combination of OM1, 2 and 3), it's convenient to remember that ASC-OMC_A is far field and OMC_B is near. It seems that that's the case as of now.
You always want to use the mirror closer to the OMC (like OM3) to center the far QPD (OMC_A) and the far mirror (like OM1) to center the closer QPD. Iterate this and things converge.
Even when you only see the beam on one QPD, you can use the above rule. For example, if your beam is on OMC_B but not on OMC_A, you will scan OM3 by a large amplitude to find the bean on OMC_A. It doesn't matter if you lose the beam on OMC_B at this point, next thing to do is to scan OM1 to regain the beam on OMC_B. Iterate and you'll find the beam on both of the QPDs at some point.
Jenne D, Jennie W
Jenne noticed that our AS_A_YAW offset was on in the current state (PREP_FOR_LOCKING). We don't want this offset on currently as we are moving to a different spot on the OFI for which we need the picomotors, therefore we my need to rethink AS_A and B alignment offsets.
We turned this offset and the other AS_A and AS_B oiffsets off.
This is the only one I could find set in the ISC_LOCK guardian so I commented this line out.
elif self.counter==9 and self.timer['LoopShapeRamp']:
#ezca['ASC-AS_A_DC_YAW_OFFSET'] = -0.15 J Wright commented this out on 6th June 2024.
ezca.switch('ASC-AS_A_DC_YAW', 'OFFSET', 'ON')
I reloaded the ISC_LOCK guardian.
One of the three pressure spikes from yesterday was the lock loss from 10W, others were from 60W.
1st plot shows the worst one in terms of the pressure spike (the last one at 6/7 1:12:35 UTC, 60W).
This doesn't tell us the cause of the pressure spike, that requires different investigations.
Anyway, the 2nd plot shows the lockloss from today after Sheila, Jennie and TJ moved to a better alignment.
SR2 doesn't move much over the power range from 2W to 25W, and ASC-AS_C_NSUM doesn't show the sign of clipping. Everything looks saner and thus better.
The only concern is if this "better" alignment will still burn things in HAM6. We haven't observed pressure spikes from 2W lock losses with this alignment today but that only happened twice.
To clarify, "bad" alignment that produced pressure spikes was only for Thursday, and the "better" alignment today is supposed to be the same as or at least quite similar to pre-Thursday. Since we haven't had any pressure spikes with pre-Thur. alignment, I doubt that we'll see any pressure spikes with this alignment either.
However, picos had to be moved by large amount to go from pre-Thur. to Thur. and then to Fri. alignment. Even though Fri. alignment should be clouse to pre-Thur., we expect some not-so-large difference in the WFS sled centering, thus in the HAM6 alignment.
Out of caution, we'd like to see that losing lock at 10W won't cause pressure spikes before proceeding further. We chose 10W, because a pressure spike was observed at that power with a loss of lock with a bad (Thu.) alignment. If we don't see any spike for many lock losses with current (Fri.) alignment it would be safe to proceed.
Since automated DC lock won't work for this "good" alignment yet (that problem existed before Thursday and it seems to persist today), Sheila and I asked Tony to lock the IFO, bring it to 10W with RF, kill lock and observe pressure. If there's a spike he will stop, otherwise he will repeat.
We'll reconvene tomorrow.
Attached is an annotated HAM6 picture.
Also, plots with more piranis showing all three pressure spikes was posted in alog 78327.
Since FS seems to have worked, potential point that could have made pressure spikes are: