TITLE: 03/14 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Currrently Observing at 148Mpc and have been Locked for 5 hours. The range is still low and fluctuating a bit, but it's not as bad as it was before I made the changes to the multiple phases (83367). Secondary microseeism is going back down, and the wind is low.
LOG:
23:30UTC Relocking
23:50 NOMINAL_LOW_NOISE
- Ran SQZ ANG ADJUST to check if the SQZ servo was in a location to pull the ADF phase away
23:59 Observing
02:55 Out of Observing to check sqz angle
- Ran SQZ ANG ADJUST and had to adjust some phases (83367)
03:13 Back into Observing
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
21:21 | ISC | Camilla | OpticsLab | y(local) | Beam dump positioning for PM1 | 22:30 |
21:47 | SEI | Jim | LVEA | n | SEI rack part grab | 21:50 |
22:46 | ISC | Mayank | Opt lab | Yes | ISS PD array | 00:48 |
22:58 | ISC | Keita, Rahul | Opt Lab | Yes | PM1 optic testing | 00:03 |
23:38 | ISC | Siva | OpticsLab | y(local) | ISS PD array work | 00:48 |
23:53 | ISC | Camilla | OpticsLab | y(local) | Checking in | 00:01 |
We are Observing at 150Mpc and have been Locked for 4 hours.
We just got back into Observing after I took us out to touch up some sqz phases. This morning Camilla and Sheila had found (83350) that the SQZ servo was pulling the ADF phase away and causing large range drops. The same thing also happened the lock after that.
When we first got to NOMINAL_LOW_NOISE, I tried running SQZ ANG ADJUST to check if we were in a spot where the ADF phase would end up getting pulled away. At the location where the squeezing was best, the ADF sqz angle was pretty good, and the demod error was near 0 (ndscope1). I didn't adjust anything because of this, just went back into FDS and then into Observing. I planned on monitoring the ADF phase angle and then do these tests again once we were thermalized.
Over the course of thermalization, the same issue with the ADF phase was happening, although luckily the servo wasn't taking the phase in circles like it had been doing the previous couple of locks (ndscope2). Once we were thermalized, I took us out of Observing and ran SQZ ANG ADJUST. From that I saw that the best location for the sqz phase was now very close to a turning point (ndscope3). To move away from this location, I adjusted SQZ-ADF_VCXO_PLL_PHASE from 25 down to 10, and then adjusted SQZ-ADF_OMC_TRANS_PHASE from -107 to -117 to get SQZ-CLF_REFL_RF6_PHASE_PHASEDEG to oscillate around 0. The yellow BLRMS looks a lot better now and our range has gone up by 10Mpc, but my changes did make the purple BLRMS worse by a bit (ndscope4, ndscope5). I accepted the phase changes in SDF and we went back into Observing (SDF diffs).
Janos, Gerardo, Dave:
The slope trip level for some LVEA gauges was reduced from 1.0e-10Torr to 3.0e-12Torr such that if an event similar to this morning were to happen again VACSTAT would alarm on it.
Trending the SLOPE channels over the past week showed that BSC3's gauge (PT132) cannot be lowered without causing many false positive alarms. Attachments show trend and which gauges now have the lower trip level.
TITLE: 03/13 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Calibration and commissioning time this morning, but we have also been struggling a bit with relocking. We aren't entirely sure what particular problem is causing the lock losses on acquisition, but the elevated wind and useism aren't helping. We are now on DC readout though, so we're making progress.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
16:01 | PEM | Robert | LVEA | n | Setup shaker | 16:22 |
16:22 | IAS | Jason | LVEA | n | FARO info grab | 16:38 |
16:31 | PEM | Robert | EX | n | Shaker setup | 16:59 |
17:57 | SEI | Jim, Mitchell | LVEA,FCES | n | Check on racks | 17:58 |
21:21 | ISC | Camilla | OpticsLab | y(local) | Beam dump positioning for PM1 | 22:30 |
21:47 | SEI | Jim | LVEA | n | SEI rack part grab | 21:50 |
22:46 | ISC | Mayank | Opt lab | Yes | ISS PD array | ongoing |
22:58 | ISC | Keita, Rahul | Opt Lab | Yes | PM1 optic testing | ongoing |
TITLE: 03/13 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 25mph Gusts, 16mph 3min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.34 μm/s
QUICK SUMMARY:
At ENGAGE_ASC_FOR_FULL_IFO and working on relocking with lots of lower state locklosses. These seem to be due to various causes, including high-ish winds& gusts. Secondary microseism is also a bit elevated.
I replaced the three pole, 25 amp circuit breaker which controls the line voltage for an HVAC circuit in the H2. Voltage output was 490 volts.The previous breaker was damaged by heat caused by a loose connection.
No obvious cause but there was a small change in the PRG a second or two before lock loss.
We lost lock from the calibration, so we tried to lock ALS without the linearization (some background in this alog: 83278.) An active measurement of the transfer function from DRIVEALIGN_L to MASTER out was 1 without the linearization, and -0.757 with the linearization on. So I've changed the DRIVEALIGN gain to -1.3 in the ALS_DIFF guardian when the use_ESD_linearization is set to false.
We tried this once, and it stayed locked for a DARM gain of 400, but unlocked as the UIM boosts were turning on. We tried this again but it also didn't lock DIFF, so it is now out of the guardian again.
I looked at a few more of the past ALS DIFF locks, both sucsesful and unsucsesful attempts we are saturating the ESD (either the DAC or the limiter in the linearization) in the first steps of locking DIFF. We do these steps quite slowly, stepping the darm gain to 40 waiting for the DARM1 ramp time, stepping it to 400, then waiting twice the ramp time, then engaging the boosts for offloading to L1. I reduced the ramp time from 5 seconds to 2 seconds to make this go faster. This worked on the first locking attempt, but that could be a coincidence.
We will leave this in for a while, so that we can compare how frequently we loose lock at LOCKING_ALS. In the last 7 days we've had 48 LOCKING_ALS locklosses, and 19 locklosses from NLN, so roughly 2.5 ALS locklosses per lock stretch.
Since the time of this alog, around 19 UTC on March 13th, we've had 68 locking_ALS locklosses and 12 NLN locklosses, so about 6 locklosses per sucsesful lock. It seem though that the change to 2 seconds was never in place, and the guardian code still said 5 seconds. So this issue seems to be getting worse without any change.
Now I've loaded the change to 2 seconds, so this should be sped up after today's maintence window.
I've looked at a bunch more of these locklosses, and they mostly happen in the time when the DARM gain is ramping, less often as the boosts are coming on in L1, and 1 I saw happened while COMM was locking.
In all the cases the linearization seems to hit its limiter before anything else goes wrong.
The calibration measurement caused another lock loss at the end of the measurement (alog83351)
Simulines start:
PDT: 2025-03-13 08:36:53.447733 PDT
UTC: 2025-03-13 15:36:53.447733 UTC
GPS: 1425915431.447733
End of script output:
2025-03-13 15:58:59,923 | INFO | Drive, on L2_SUSETMX_iEXC2DARMTF, at frequency: 42.45, and amplitude 0.41412, is fin
ished. GPS start and end time stamps: 1425916738, 1425916753
2025-03-13 15:58:59,923 | INFO | Scanning frequency 43.6 in Scan : L2_SUSETMX_iEXC2DARMTF on PID: 795290
2025-03-13 15:58:59,923 | INFO | Drive, on L2_SUSETMX_iEXC2DARMTF, at frequency: 43.6, is now running for 23 seconds.
2025-03-13 15:59:01,039 | INFO | Drive, on DARM_OLGTF, at frequency: 1083.3, and amplitude 1e-09, is finished. GPS st
art and end time stamps: 1425916738, 1425916753
2025-03-13 15:59:01,039 | INFO | Scanning frequency 1200.0 in Scan : DARM_OLGTF on PID: 795280
2025-03-13 15:59:01,039 | INFO | Drive, on DARM_OLGTF, at frequency: 1200.0, is now running for 23 seconds.
2025-03-13 15:59:02,168 | INFO | Drive, on L1_SUSETMX_iEXC2DARMTF, at frequency: 11.13, and amplitude 13.856, is fini
shed. GPS start and end time stamps: 1425916735, 1425916753
2025-03-13 15:59:02,168 | INFO | Scanning frequency 12.33 in Scan : L1_SUSETMX_iEXC2DARMTF on PID: 795287
2025-03-13 15:59:02,169 | INFO | Drive, on L1_SUSETMX_iEXC2DARMTF, at frequency: 12.33, is now running for 25 seconds
.
2025-03-13 15:59:07,162 | ERROR | IFO not in Low Noise state, Sending Interrupts to excitations and main thread.
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L2_CAL_EXC
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L3_CAL_EXC
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:CAL-PCALY_SWEPT_SINE_EXC
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L1_CAL_EXC
2025-03-13 15:59:07,163 | ERROR | Aborting main thread and Data recording, if any. Cleaning up temporary file structu
re.
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:LSC-DARM1_EXC
ICE default IO error handler doing an exit(), pid = 795246, errno = 32
PDT: 2025-03-13 08:59:11.496308 PDT
UTC: 2025-03-13 15:59:11.496308 UTC
GPS: 1425916769.496308
Closes FAMIS 26034, last checked in alog83240
The script reports high freq noise is elevated for these BSCs.
All the HAMs noise looks to have reduced except for HAM8, particularly a peak at just over 1 Hz in H3 and H1 looks reduced on almost all of the HAMs. HAM8s mid freq looks overall noisier and the high freq V3 peak looks elevated (just over 100Hz).
This happened at the end of the calibration measurement. This also happened last week (alog83210).
The last 3 calibration measurements have ended in locklosses, todays, saturdays and last thursdays (these link to the lockloss tool webpages). The 3 locklosses look fairly similiar but looking at the ETMX signals, todays seemed a little stronger of a lockloss? Compared to Saturday and last thursday.
Last thursdays lockloss alog83239
Looking at the same channels that Vlad did in 82878 and Ryan's above ETMX channels, the 42Hz growing wobble in DARM/ETMX seems to start as soon as the DAM1_EXC is ramped on to it's full amplitude, plot attached. The 42Hz is seen as an excitation into H1:SUS-ETMX_L2_CAL_EXC_OUT_DQ, plot attached. Maybe the ETM is not fully stable at that frequency?
Looking at a successful calibration sweep Feb 22nd 82973, There was no instability when the 42Hz oscillation was turned onto ETMX L2, plot attached.
We had a vacuum glitch in the LVEA at 01:57 Thu 13mar2025 PDT which was coincident with a lockloss. The glitch was below VACSTAT's alarm levels by an order of magnitude, so no VACSTAT alert was issued.
The glitch is seen in most LVEA gauges, and took about 10 minutes to pump down.
Attached plots show a sample of LVEA gauges, the VACSTAT channels for LY, and the ISC_LOCK lockloss.
Adding the lock loss tag and a link to the lock loss tool - 1425891454
There's no blatantly obvious cause. The wind was definitely picking up right before the lock loss, which can be seen in many of the ASC loops, but I'm not sure it was enough to cause the lock loss.
This may be a what-came-first the LL or the VAC spike scenario but looking at the past 3 hour trend of H1:CDS-VAC_STAT_LY_Y4_PT124B_VALUE attached, it does not go above 3.48e-9. However 3 seconds before the lockloss, it jumps to 3.52e-9 attached. This seems suspiciously like whatever caused the VAC spike came before the lockloss.
There is a LL issue open to add a tag for this: #227, as it was previously seen in 82907.
The only possible plausible vacuum glitch without the laser hitting something first, is IP-glitch. Gerardo is looking into this now. Otherwise, I would say the laser hit something, which didn't cause lockloss right away (only 3 seconds later), but - obviously - caused pressure spike. The rising wind is just too big of a coincidence to me..
Corner RGA caught small change in H2 and N2 at the time of the vacuum glitch. AMU 2 delta = 5.4e-10 amp, AMU 28 delta = 1.61e-10 amp
Attached is a screenshot of RGA scan just before the vac glitch (1:57:02 AM 3/13/25), and second screenshot is ~10 seconds after. Top trace is the 0-100AMU scan at that time, bottom trace is the trend of typical gas components (AMU 2, 18, 28, 32, 40, 44), over ~50 minutes. Vertical line on bottom trace corresponds to time RGA was collected
RGA is a Pfeiffer Prisma Plus, 0-100AMU, with 10ms dwell time, EM enabled with 1200V multiplier voltage. Scans run continuous 0-100AMU sweeps
Main ion pumps reacted to the "pressure spike" after it was noted by other instruments such as the vacuum gauges, see attached plot, first one.
The second plot shows the different gauges located at the corner station, the "pressure spike" appears to be noted first by two gauges PT120 (gauge on dome of BSC2) and PT152 (gauge located at the relay tube). The amplitude of the "pressure spike" was very small, signature only was noted at the corner station, not noted at Mids or Ends.
Two of the ion pumps at the filter cavity tube responded to the "pressure spkie", see third attachment.
Also, the gauges located on the filter cavity tube noted the spike, including the "Relay Tube".
As a potential optic for the new TT for the in-vac POPX WFS in HAM1, we tested the transmission of what is supposed to be the old ZM2 mirror from O3 days as a function of AOI.
The results summarized below doesn't make sense because it's least transmissive at 0 deg AOI and it's not HR at 45 deg (3.3% transmission). It's supposed to be E1700103 according to the label but with a handwritten note saying "verify??? doesnt look the same" or something like that. Anyway, if it is indeed E1700103, it should be R>99.99% @ 1064nm for both P and S at 45 +- 5 deg AOI.
This mirror is probably usable for POPX in that the transmission is not disastrously big at the proposed AOI (~30deg or so?) but this is a mystery optic as far as I'm concerned. If the mirror in ZM1 is better, we'll use it.
AOI (deg) | power in transmission (mW) | power transmission in % |
0 | 0.05 | 0.03 |
10 | 0.08 | 0.04 |
15 | 0.09 | 0.05 |
20 | 0.11 | 0.061 |
25 | 0.16 | 0.089 |
30 | 0.28 | 0.16 |
35 | 0.44 | 0.24 |
40 | 1.23 | 0.68 |
45 | 5.94 | 3.3 |
Other things to note:
We mounted the mirror in a class A siskiyou mirror holder on a dirty rotational stage with a clean sheet of foil inbetween so we can easily change the AOI in YAW.
The apparatus was NPRO - BS (to throw away some power) - HWP - PBS cube transmission (to select P-pol) - steering - ZM2 optic transmission - power meter.
The beam height between the steering mirror and the ZM2 optic was leveled, then ZM2 was adjusted so the beam retroreflects within +-2mm over 19 inches, and this became our AOI=0 reference point with uncertainty of ~+-2mm/19inches/2 rad ~ +-0.1deg. We used the indicator on the rotational stage to determine the AOI, and overall AOI error should have been smaller than 1 deg.
Before/after the measurement, the power impinging the HR side of the mirror was 180.3mW/176.1mW. Since the transmitted power only had 2-digits precision at best, I used 1.8e2 mW as the input power. The ambient light level was less than 10uW and is ignored here.
Another obvious candidate is ZM1 (again from O3 days). The mirror was not removed from TT after deinstallation though. We'll see if we can roughly determine the AOI while the mirror is still suspended, as it's a hassle to remove it from the suspension and there's a risk of chipping.
We noticed that the range was dropping even lower than our already low ~145Mpc. There was a lot of low frequency noise that was creeping up. Sheila suggested that we look at the PIs and sure enough, PI31 had started to slowly creep up the same time as the range degredation (see attached). The SUS_PI guardian node will turn on the damping at 3 according to the H1:SUS-PI_PROC_COMPUTE_MODE31_RMSMON channel and turn it back off when it gets below 3. For now we just tried changing that threshold to 0 to all for continuous damping. This let the mode damp down completely and it brought our range back up.
We should think more about damping this down lower and having two thresholds, an on an off.
Changed the threshold to 1. At 0 the node would continuously search for a new phase thinking that it needed to damp it further.
The first attached screenshot shows the spectrum around 10.4 kHz (edited to fix typo), the mode which rang up and caused this issue is at 10428.38 Hz, previously identified as being in the y arm (Oli's slides).
The behavoir seemed pretty similar to what we saw a few weeks ago, 82961, with broad nonstationary noise in DARM.
The downconversion doesn't seem to be caused by the damping, it started to have a noticable impact on the range while the damping was off, and when the damping came on and reduced the amplitude of the mode the range improved.
In the ndscope screenshot attached you can see the range that we are using on the DCPDs ADC, this is a 20 bit DAC so it would saturate at 524k counts, when this PI was at it's highest today it was using 10% of the range, the DARM offset takes up about 20% of the ADC range.
The third attachment is the DARM spectrum at the time of this issue, as requested by Peter. As described in 83330 our range is decreasing with thermalization in each lock, the spectrum shows the typical degredation in the spectrum in these last few days since the SQS angle servo has been keeping the squeezing more stable. The excess noise seen when the PI was rung up has a similar spectrum to the excess noise we get after thermalization.
Also, for some information about these 10.4kHz PI modes, see G1901351
Jeff pointed me to 82686 with the suggestion to check the channels that have different digital AA.
The attached plot shows DCPD sum, and two alternatives, in red is H1:OMC-DCPD_16K_SUM1_OUT_DQ which has a high pass to reduce the single precision noise and no digital AA filters (and some extra lines), SUM2 has 1 digital AA filter and the high pass, and DCPD sum is the darm channel and has 2 digital AA filters and no high pass. The broadband noise is the same for all of these, so digital aliasing doesn't seem to be involved in adding this broadband noise.
WP9880
I removed the O3 ZM2 tip tilt from HAM5 and moved the SRM heater optics to be used in the upcoming temporary laser setup for OFI alignment.
Summary of activities:
just for info - the optic used for ZM2 (HTTS, HAM5) optic is E1000425 - currently stored at LHO.