oli.patane@LIGO.ORG - posted 13:43, Thursday 31 October 2024 (80977)
Lockloss
Lockloss @ 10/31 18:35UTC due to small earthquake, high microseism, and low CARM actuation range(80969)
Lockloss @ 10/31 18:35UTC due to small earthquake, high microseism, and low CARM actuation range(80969)
Thu Oct 31 10:07:19 2024 INFO: Fill completed in 7min 16secs
Jordan confirmed a good fill curbside.
We re-enabled FC beam spot control, by enabling inputs to the H1:SQZ-FC_ASC_INJ_ANG_{P,Y} filter banks. Seemed to just work (see attached screenshot) - moves ZM3 to control the spot on the green QPD in filter cavity transmission (SQZ-FC_TRANS_A on SQZT8). Have not yet checked the centering, last done in May 2024 (see 77942 , 69276 , 69374 , 73777).
QPD offsets are set in the FC_ASC_INMATRIX filter banks, last column and row.
I set the flag in sqzparams.py (use_fc_beamspot_control = True), and reloaded SQZ_FC guardian so it will be used in the next locks.
Disabled FC beam spot control (use_fc_beamspot_control = False) and loaded SQZ_FC guardian. We can try that again later, beam should be closer to centered now.
FC had a hard time locking in this past lock (ending ~0100 UTC), so that lock didn't have squeezing injected. We first used sliders to set ZMs2-3 and FC1-2 back according to M1_DAMP_{P/Y}_INMONs, to a time during the last lock. FC alignment was still bad with slides set back (bad green tem00 trans, green vco not catching the lock stably).
Then see the attached scope for the QPD alignment that worked. Overall what we should have done: 1) bringing SQZ_MANAGER through DOWN first (I think that fixed something). 2) Setting ZM3+FC1 according to RLF QPDs in HAM7, then set FC2 according to green trans QPD (FC_QPD_TRANS_A) on SQZT8. Biggest move was moving FC2 Y by about +20 on the slider, which is a really big change for this one. With setting SUS back based on top mass osems, then green FC_REFL looked bad, and FC couldn't hand off the IR lock successfully. With the alignment looking good on all QPD's, the FC alignment looks stable (both FC_GR_TRANS and FC_GR_REFL), and I think FC should lock next time.
Yesterday we went back to the higher range CARM configuration, in which the fast/IMC locklosses first appeared 80938. Overnight we had two observing locklosses tagged with the IMC, neither of which had glitches visible in the FSS fast mon channel, 1414411965 (1st attached screenshot), and 1414381791.
This morning I changed the gains by 6 dB in the other direction, so we further limit the range available to the frequency actuator. (CARM sliders and ndscope) Our microseism is around 5um/second right now, slightly above the 90th percentile, but there was no wind and no earthquakes this morning. At times the IMC splitmon was reaching -8V, it's range is +/- 10V so this is about as far as we can go in this direction, and might be kind of marginal.
We've since had one lockloss 1414426971, this is tagged as commissioning because we were in commissioning time at the moment, but no one was actively doing anything at the time so this wasn't caused by commissioners. There was a glitch in the FSS fast mon for this lockloss, although it was below the threshold for the tag to be applied, and around the time of that glitch the drive to the ISS AOM becomes noisy. We were not saturating the IMC splitmon at the time of the lockloss so that doesn't seem to be the cause.
I've removed this, so now we will be back to the gains we've been with for the last several weeks, and most of O4.
Lockloss @ 10/31 16:01 UTC after almost an hour locked
18:13 Just got to NLN, staying out of Observing for Commissioning
All models on h1seih45 glitched at 07:09:46 PDT (14:09:46 UTC). This appears to have broken the lock and caused guardian to start an initial alignment.
Symptoms are:
The models appeared to have resumed normal operation following the glitch, no restarts were needed. At the time of writing H1 is locked with a range of 160MPc.
At the time of the glitch we had been relocking and at TRANSITION_FROM_ETMX. There seem to have been no issues at all from this - the initial alignment was automatically started after trying to relock took us into CHECK_MICH_FRINGES for the third time during the relock period, and went smoothly and without issue.
HAM4 and HAM5 GS13 signals following the glitch:
There are 8 Dolphin IPC channels which are sent by h1seih45 models (h1isiham4 and h1isiham5). They are all received by h1seiproc on h1oaf0. h1seiproc runs at 4kHz.
All 8 receive channels saw 4096 errors at the time of the glitch.
rcv model type snd host snd model ipc channel
h1seiproc PCIE h1seih45 h1isiham4 H1:ISI-HAM4_CPS_X_IPC_OUT
h1seiproc PCIE h1seih45 h1isiham4 H1:ISI-HAM4_CPS_Y_IPC_OUT
h1seiproc PCIE h1seih45 h1isiham4 H1:ISI-HAM4_SUSPOINT_SR2_L_OUT_IPC
h1seiproc PCIE h1seih45 h1isiham5 H1:ISI-HAM5_CPS_X_IPC_OUT
h1seiproc PCIE h1seih45 h1isiham5 H1:ISI-HAM5_CPS_Y_IPC_OUT
h1seiproc PCIE h1seih45 h1isiham5 H1:ISI-HAM5_SUSPOINT_OFI_L_OUT_IPC
h1seiproc PCIE h1seih45 h1isiham5 H1:ISI-HAM5_SUSPOINT_SR3_L_OUT_IPC
h1seiproc PCIE h1seih45 h1isiham5 H1:ISI-HAM5_SUSPOINT_SRM_L_OUT_IPC
TITLE: 10/31 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 5mph Gusts, 3mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.47 μm/s
QUICK SUMMARY:
Looks like we just finished an initial alignment and are starting to relock
H1 called for assistance at 13:13 UTC when SRM watchdogs tripped during the SRC alignment steps of initial alignment. After resetting the WDs, I just let Guardian try SRC align again, but SRM tripped during ACQUIRE_SRY. I reset the WDs again and took ALIGN_IFO to DOWN to manually align SRM a bit (didn't get too much improvement on the AS_A fringing, but some), and had Guardian try again. This time, there were no issues, and initial alignment finished without issue.
H1 is currently relocking and just got through DRMI.
TITLE: 10/31 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Currently relocking, but we've had a few lock losses after DRMI and one at engage_asc_for_full_ifo. We've just made it past that point again.
Lockloss 0349
Averaging Mass Centering channels for 10 [sec] ...
2024-10-30 14:34:04.266740
There are 14 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 2 DOF X/U = -0.755 [V]
ETMX T240 2 DOF Y/V = -0.707 [V]
ETMX T240 2 DOF Z/W = -0.473 [V]
ITMX T240 1 DOF X/U = -1.549 [V]
ITMX T240 1 DOF Y/V = 0.328 [V]
ITMX T240 1 DOF Z/W = 0.452 [V]
ITMX T240 3 DOF X/U = -1.611 [V]
ITMY T240 3 DOF X/U = -0.797 [V]
ITMY T240 3 DOF Z/W = -1.997 [V]
BS T240 1 DOF Y/V = -0.404 [V]
BS T240 3 DOF Y/V = -0.317 [V]
BS T240 3 DOF Z/W = -0.502 [V]
HAM8 1 DOF Y/V = -0.436 [V]
HAM8 1 DOF Z/W = -0.758 [V]
All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF X/U = -0.04 [V]
ETMX T240 1 DOF Y/V = -0.066 [V]
ETMX T240 1 DOF Z/W = -0.098 [V]
ETMX T240 3 DOF X/U = -0.055 [V]
ETMX T240 3 DOF Y/V = -0.111 [V]
ETMX T240 3 DOF Z/W = -0.058 [V]
ETMY T240 1 DOF X/U = 0.015 [V]
ETMY T240 1 DOF Y/V = 0.136 [V]
ETMY T240 1 DOF Z/W = 0.192 [V]
ETMY T240 2 DOF X/U = -0.069 [V]
ETMY T240 2 DOF Y/V = 0.187 [V]
ETMY T240 2 DOF Z/W = 0.029 [V]
ETMY T240 3 DOF X/U = 0.206 [V]
ETMY T240 3 DOF Y/V = 0.04 [V]
ETMY T240 3 DOF Z/W = 0.128 [V]
ITMX T240 2 DOF X/U = 0.129 [V]
ITMX T240 2 DOF Y/V = 0.265 [V]
ITMX T240 2 DOF Z/W = 0.229 [V]
ITMX T240 3 DOF Y/V = 0.124 [V]
ITMX T240 3 DOF Z/W = 0.102 [V]
ITMY T240 1 DOF X/U = 0.083 [V]
ITMY T240 1 DOF Y/V = 0.084 [V]
ITMY T240 1 DOF Z/W = -0.063 [V]
ITMY T240 2 DOF X/U = 0.05 [V]
ITMY T240 2 DOF Y/V = 0.182 [V]
ITMY T240 2 DOF Z/W = 0.1 [V]
ITMY T240 3 DOF Y/V = 0.072 [V]
BS T240 1 DOF X/U = -0.174 [V]
BS T240 1 DOF Z/W = 0.082 [V]
BS T240 2 DOF X/U = -0.099 [V]
BS T240 2 DOF Y/V = 0.002 [V]
BS T240 2 DOF Z/W = -0.126 [V]
BS T240 3 DOF X/U = -0.213 [V]
HAM8 1 DOF X/U = -0.286 [V]
Reran all O4b locklosses on the locklost tool with the most up to date code, which includes the "IMC" tag.
I used the lockloss tool data since the emergency OFI vent (back online ~23rd August) until today and did some excel wizardry (attached) to make the two attached plots, showing the number of locklosses per day tagged with IMC and without tag IMC. I made this plots for just locklosses from Observing and all NLN locklosses (can include commissioning/maintenance times), including:
Attaching plots from zooming in on a few locklosses:
| Time | Tags | Zoom Plot | Wide Plot |
| 2024-10-31 12:12:26.993164 UTC (IMC) | OBSERVE IMC REFINED | annotated plot | plot |
| 2024-10-30 12:16:33.504883 UTC (IMC) | OBSERVE IMC REFINED | annotated plot | plot |
| 2024-10-30 16:09:21.548828 UTC (Normal) | OBSERVE REFINED OMC_DCPD | N/A | annotated plot |
The 10-31 zoom plot notes the framerates of the channels: ASC, REFL and NPRO_PWR are 2kHz and GS13 is 4kHz, the others are 16kHz.
Since September 18th we've had 21 locklosses from NLN tagged as FSS_OSCILLATION, of these 20 also had the IMC tag. Since September 12th, we've had 49 locklosses from NLN tagged IMC, so roughly 40% of these IMC locklosses have the FSS oscillation tag, since the NPRO was swapped we don't have any tagged with FSS_OSCILLATION.
(Reminder, the FSS_OSCILLATION tag is an old tag, intended for a different problem, but it tags times where the FSS fast mon goes above a treshold.)
Updated plot attached of NLN locklosses tagged with and without the IMC tag.
J. Freed
I followed Step 1 of the double mixer test plan in T2400327 under "1. Characterize frequency references."
PNSPIDACTestStand.pdf Shows the results in dBc/Hz. And PNSPIDACTestStandrad.pdf shows the result in rad/sqrt(Hz).
Besides the 80MHz standard reference there were 2 devices under test, the IFR 2023A and the SRS SG382. These two devices went under 3 tests involving different combinations of
1. Timing to a 10MHz frequency standard produced by the 80MHz Standard. (Time Standard)
2. PLL locking to a different 80MHz frequency ref through the tune in port. (Lock)
During this process it was assumed that the 80MHz standard and the added 80MHz ref would have a simmilar noise profile as they come from a simmialar OCXO.
A 4th test was added to the Time Standard/No Lock test which used the SR785s internal high pass filter (-3dB at 0.16Hz) to remove some of the DC components from the test. This is because the results of the initial test were inconclusive as the noise floor of the SR785 was too high. The noise floor was too high because there was a DC signal that caused the input range of the SR785 to be about 20dB. The high pass filter, removed the DC signal, lowering the noise floor at the cost of signals below about 0.2Hz not being accurate. We believe this is caused by the fact that while the PLL locking can be controled (tries to lock signals to destructivly interfere so to lower output power for Phase noise tests), there was no control by the Time Standard on phase differences between oscilators. As Time Standard/ No lock, showed the best results above 0.06Hz and is closest to the SPI set up design, this fact may be important when doing futher tests.
Between IFR and SRS, the SRS showed better phase noise performance below 7kHz. While IFR showed better performance above 7kHz. As such, SRS shows more promise in SPIs phase noise range of interest 0-4096Hz.
After a talk with Jeff what is really important to show is the TIme Standard/Lock which is the 80 MHz OCXO, IFR and SRS which we believe is most represntative of phase noise
These are, one at a time, locked to another 80 MHz REF OCXO. In this it was assumed that the 80 MHz OCXO and the 80 MHz REF OCXO would produce the same phase noise, as such contribute equally to the output phase noise. As such the output was manually halved to represent the only the 80 MHz OCXO. As stated before these measurments also had a Time Standard (sometimes called time sync) where the 80 MHz OCXO is synced to the site's 1PPS. And the IFR and SRS have a 10MHz signal time standard signal that was produced by the 80 MHz OCXO divided by 8 and sent into the back of the devices.
PNSPIDACTestStandrad(sync_lockOnly).pdf Shows only the results of the Time Standard/Lock, which we believe to be the most represntative of the phasenoise of these devices.
Since it was assumed that the two OCXOs would have the same phase noise, this also makes our noise limit equal the OCXO. As such, it can be seen from this graph that we are at the noise limit in measuring the phase noise of the IFR and SRS between 0.1Hz-10Hz.
Continuing on from Jenne's observation that there are still glitches in the new NPRO, I've tried to make a plot we can use to compare the glitch rate in the new and old NPROs, using the NPRO_PWR channel instead of the FSS channel which isn't available for the new NPRO.
I've used a 3 hour stretch of observing time for the old NPRO, and a three hour stretch before the time when Jenne made a plot in 80837. The ISS is not on for the new NPRO time, which is probably why the intensity is flcutuating and making it harder to see the small steps in power that are the glitches we are looking for. In the second panel, I've plotted the data high passed with a 0.05 Hz butterworth, this helps to show the glitches, although not perfectly (in either case). Based on this, the glitches look to be happening at a roughly similar rate, although somewhat less with the new NPRO.
This script in in sheila.dwyer/DutyCycle)4/dutycycleplots/PSL_glitches.py
Attaching another plot, showing that comparing our old NPRO to LLO during an observing stretch that started at midnight UTC time on Oct7th, LLO has no similar glitches.
Here's the same plot, but using a time from last night when the PSL environmental controls were off. There are still gltiches, but fewer.
These plot tiles show runs of Sheila's code looking for PSL power glitches on several days before / after the suspect date around Sept 12.
There's not a clear correlation between the glitches and locklosses. While maybe there's more glitches after Sept 12 (bottom row), the glitches don't consistently correlate with locklosses? Sept 14 is a good example of this: lots of glitches, the IFO stays locked through many of them.
2nd plot here shows overnight again with the swapped new laser. There are still glitches (though potentially less).
The PSL-PWR_NPRO_OUT_DQ channel seems to not be connected at LLO, which explains why the comparison plot a few comments above makes it look like L1 PSL is so much quieter than H1.
Adding screenshot of the NPRO power glitches over the past day. There are still glitches with the new laser -- not all glitches correspond to locklosses, but some do.
Sheila, Vicky, Camilla
We have turned back on the SQZ angle servo using the ADF at 322Hz. Last briefly tried while testing ADS alignment in ADS in 80194. Turned on ADF and used 'python setADF.py -f 322'. Then set H1:SQZ-ADF_OMC_TRANS_PHASE to get H1:SQZ-ADF_OMC_TRANS_SQZ_ANG close to zero and checked by stepping the SQZ angle that there is a zero crossing in the ADF measured SQZ angle, plot attached.
The servo adjusts the SQZ angle (H1:SQZ-CLF_REFL_RF6_PHASE_PHASEDEG) via keeping the ADF measured angle (H1:SQZ-ADF_OMC_TRANS_SQZ_ANG) at zero. Setpoint can be adjusted using the ADF phase (H1:SQZ-ADF_OMC_TRANS_PHASE).
Tagging Detchar: ADF is now on at 322Hz. It was turned all the way off in 79573 by Alan. We can adjust the frequency 50-500Hz if there is a better place for a line.
Note to operators: if you want to run SCAN_SQZANG, the ADF servo will now overwrite the sqz angle. So BEFORE going back to FREQ_DEP_SQZ you'll want to tweak H1:SQZ-ADF_OMC_TRANS_PHASE (via sqz overview > ADF) to make H1:SQZ-ADF_OMC_TRANS_SQZ_ANG close to zero. Or you can tweak H1:SQZ-ADF_OMC_TRANS_PHASE (via sqz overview > ADF) until the SQZ BLRMs/ DARM is best.
Trends of the ADF servo stabilizing the SQZ angle overnight. Looks good: the ADF SQZ ANGLE servo can hold the maximum squeezing level throughout the lock! Last night was running with the ADF SQZ angle servo + SQZ-IFO AS42 ASC together.
In the first lock of the screenshot, the ADF SQZ ANGLE servo is not yet running, and the squeezing level drifts quite a bit (~0.5-1 dB in ~2 hours, and ends up un-optimal). In the last 2 locks, the ADF SQZ ANGLE servo is running and successfully stabilizes the SQZ angle, though the 2 locks from last night stabilize at different SQZ angles (weird?). Note SQZ ASC is running in both of these locks, so it seems like ASC + ADF SQZ ANG servo work well when used together.
Naoki looked at sqz trends with/without the ADF servo before in LHO:75000. Looking at sqz trends for yesterday, the ADF servo stabilized the SQZ angle in the first ~25 minutes. Then over the first ~2 hours, the ADF servo needed to move the CLF_RF6 demod phase by 5-10 degrees to hold the SQZ angle stable. This implies something like, the optimal injected squeezing angle changed by about 2-5 degrees during IFO thermalization.
Also noting a reference to LHO:77292, where Naoki does an On/Off test with the ADF line at 322 Hz.
Checked against the 68139 list, can see that 322Hz is a good frequency for CW. We will look at trying to add this ADF line to the _CLEAN or _NOLINES subtractions.
