Ryan C taking over for the last 4 hours but here's what happened in the first 4:
After issues getting the IMC locked during SRC_ALIGN, we manged to automatically get to NLN, Observing at 16:04 UTC.
SQZ Tuning: Dropped out of OBSERVING 17:14-17:26 UTC
Calibration Sweep: Dropped out of OBSERVING 18:34 UTC - Lockloss 19:05 UTC
Lockloss (alog 80760): FSS Caused lockloss during simulines.
Timing glitch followup (Dave): While yesterday's EY time glitch had no consequences. Dave said that it has been glitching by quite erratic numbers lately and has recommended another power supply swap (like the EX one recently).
Fast Shutter Guardian Glitch Followup: Dave found that the guardian fast shutter malfunction yesterday was caused because there was a 5s delay in data-grabbing but why that happened is unknown. He will post an alog about this soon.
Ran the usual calibration sweep following the wiki. IFO was thermalized (locked for 2.5 hrs but ~3hrs since MAX_POWER. Monitor Attached. Times are in GPS.
Broadband
Start: 1413398319
End: 1413398630
Simulines
Start: 1413398808
End (Lockloss): 1413399944
No files written on the screen due to abort from lockloss. Here's the error message:
2024-10-19 19:05:22,079 | ERROR | IFO not in Low Noise state, Sending Interrupts to excitations and main thread.
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:LSC-DARM1_EXC
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L3_CAL_EXC
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:CAL-PCALY_SWEPT_SINE_EXC
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L2_CAL_EXC
2024-10-19 19:05:22,080 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L1_CAL_EXC
2024-10-19 19:05:22,080 | ERROR | Aborting main thread and Data recording, if any. Cleaning up temporary file structure.
ICE default IO error handler doing an exit(), pid = 3195655, errno = 32
PDT: 2024-10-19 12:05:26.374801 PDT
UTC: 2024-10-19 19:05:26.374801 UTC
GPS: 1413399944.374801
PSL Caused Lockloss during last part of Simulines Calibration.
IMC and ASC lost lock within 5ms of one another. (plot attached).
20:21 UTC Observing
Looking at more channels around this lockloss, I'm not entirely sure if the FSS was at fault in this case. In a longer-term trend (starting 6 sec before lockloss, see screenshot), there were a handful of glitches in the FSS_FAST_MON channel, but no significant corresponding NPRO temp or output power changes at those times and the EOM drive had not reached high levels like we've seen in the past when this has caused locklosses. Zooming in (see other screenshot), it seems that the first of these channels to change is the FSS_FAST_MON, but instead of a glitch, it looks to actually be moving. Soon after, there's a drop in AS_A_DC_NSUM, I believe indicating the IFO was losing lock, then 5 ms later the IMC starts to lose lock. I suppose we're at the mercy of data sampling rates here, but this may add some more context to these kinds of locklosses.
Sat Oct 19 10:12:16 2024 INFO: Fill completed in 12min 12secs
TITLE: 10/19 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 19mph Gusts, 11mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY:
IFO was sitting in SRC_ALIGN part of initial alignment unable to lock the IMC, with MC2 saturations every few seconds.
This issue has happened before and it's becauser the LASER_PWR was not able to revert to 2W so was sitting at 10W - this is related to FSS glitching it seems from Ryan C's OWL alog 80754. Will set the power to 2W and attempt initial alignment again after the IMC locks.
What seems to be happening is FSS glitches and unlockes the IMC, but the power doesn't go down to 2 - this happens during MICH_FRINGES and MICH_BRIGHT_ALIGNING.
The solution that seems to work:
I had the FSS glitch out on me twice during this process meaning I had to repeat it, but it worked eventually. I think this is a temp solution.
H1 called for assistance at 14:00UTC, from Initial alignment timing out. When I got on it was in SR2 align, waiting for ASC to converge but the IMC keeps unlocking. DIAG_MAIN is reporting shutter_A is not open and IMC_LOCK keeps trying to open it, the log is saying the FSS is unlocking as well. The FSS is oscillating.
TITLE: 10/18 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:
H1's been doing decently over the last 12hrs even with breezy conditions, but toward the end of the shift, winds picked up more and getting into the 30mphs and we had a lockloss due to wind. But with all that about our high winds, H1 is currently making its way to begin powering up....but there was a lockloss at Powering up to 25W.
LOG:
Just had a lockloss with the IMC tag
But for this one there was another (pink) Verbal Alarm: "Guardian LOCKLOSS_SHUTTER_CHECK node in error." This was new to me, and I'm assuming this is similar to what Ibrahim and Sheila worked through earlier (see alog80749 + 1st photo) . I clicked LOAD to clear the RED guardian screen which then took the node to the Low_Arm_Power state (see 2nd photo).
TITLE: 10/18 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
IFO is in NLN and OBSERVING as of 21:19 UTC.
A few significant events:
PSL Maintenance: Sheila and Vicky re-reduced the NPRO current - alog 80746
Fast Shutter Weirdness: Upon reaching NLN, we could not go into observing because the shutter guardian was registering the shutter as closed/not working. After the initial shock, it was determined through trends that this was just a Guardian thing, rather than yet another broken toaster. alog 80749
EY Timing Glitch: Dave noticed an 800ns timing glitch with no consequences - alog 80742
Locklosses: Two PSL FSS Related Locklosses: alog 80748 and alog 80741. Both before the PSL change and none thereafter.
SQZ Adjustments: Went out of observing briefly to optimize SQZ angle and temp.
PIs: PI 24 and 31 rung up heavily today but were successfully and automatically damped, causing no locklosses.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
16:36 | SAFE | HAZARD | LVEA | YES | !!!!!LVEA IS LASER HAZARD!!!! | 03:16 |
19:57 | PSL | Vicky, Sheila | LVEA | Y | NPRO Current Revert | 20:20 |
22:34 | VAC | Janos | FCES Mechanical Room | N | Compressor Check | 23:25 |
TITLE: 10/18 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 23mph Gusts, 14mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY:
H1's doing nicely (*knock-on-wood!*) even with our steady winds outside & Ibrahim gave me a rundown of how the day went.
Vicky, Sheila
Summary: Today we learned that frequency independent anti-squeezing is a very good way to determine which sign the homodyne angle is.
Background: I've been working on using code from Vicky's repo and the noise budget repo to do some checks of a quantum noise model, this is in a new repo here.
Details about how this model is made:
The first attached plot illustrates how these models and plots are made. It starts with a no squeezing time, and an esitmate of non quantum noises from the noise budget, (dark gray, this one is from Elenna's recent run of the noise budget: 80603 ) and an estimate of the arm circulating power along with other parameters set in a quantum parameters file in the same format that is used by the noise budget. It fits the readout losses by adding a gwinc model of quantum noise with the noise budget estimate of other noises, and adjusting the readout losses of the gwinc model, this is done from 1.5-1.8kHz in this case.
Based on this readoutlosses we get a model of quantum noise without squeezing, and subtract that from the no squeezing trace to get an estimate of the non-quantum noise. This is enough different from the noise budget one that I've used that as the estimate of the non-quantum noise for the rest of the traces.
By subtracting this subtraction estimate of the non-quantum noise, it estimates squeezing in dB, and finds a median level of dB from 1.5-1.8kHz for anti-squeezing and squeezing. This should be the same with and without the filter cavity, but in this data set there is slightly more anti-squeezing in the time without the filter cavity, so I've used FIS and FIAS to estimate the nonlinear gain and total efficiency for squeezing. The nonlinear gain is translated into generated squeezing for gwinc, and the injection losses for squeezing are set so that the injection efficiency* readout efficiency = total squeezing efficieny.
With this information we can generate models for anti-squeezing and squeezing traces, but fitting the squeezing angle to minimize or maximize quantum noise. Then for the mid angle traces, the squeezing angle is fit to minimize the residual between the data and the quadrature sum of the subtraction estimate of non quantum noise and the model. We can then look at these plots and try manually changing parameter in the quantum parameter file.
Homodyne angle:
We've been stumped for a while about the excess noise we see with low frequency anti-squeezing, in 79775 I went through old alogs and see that we've had this mismatch of model with our data for a long time. Today we tried flipping the sign of the homodyne angle and see that low frequency anti-squeezing is much closer to fit both with and without the filter cavity. Compare the 2nd and 3rd attachments to see this.
We still have more work to do on this model, including adding in the additional traces near squeezing and near anti-squeezing that Camilla took, and checking if it can give us any information about arm power (it doesn't seem very useful for that), or the mode mismatches.
I neglected to mention that this is based on the nice data set that Camilla collected here: 80664, and that three is more work to be done with this, checking SRC detuning, mode mismatch, and including the +/- 10 deg data.
Sumary: seems the current (+) side of DARM is better for FDS, although it is opposite of our previous quantum noise models. But given the current sign is actually better for DARM, the model error doesn't really matter, and it's not really worth changing signs.
The wrong HD angle sign seems to be why none of our quantum noise models, despite fitting all other SQZ angles well, have ever fit FIAS properly. We will update our quantum noise models for the noise budget. Attached are some quantum noise models and DARM plots for Camilla's recent SQZ dataset lho80664.
Plots with optimal FDS (optimal fc detuning) for both signs of the homodyne angle: showing 1st just the quantum noise models without adding back non-quantum noise (NQN), and 2nd showing QN models + NQN.
Third attachment (3rd) shows a wider range of homodyne angles, from +15 deg to -10 deg. So far the code for these plots is living here.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Altogether this is making progress on the quantum noise models for the noise budget!
Summarizing updates and what we're learning:
Vicky, Sheila
Based on the fit of total squeezing efficiency and nonlinear gain (which is based on subtracted SQZ and ASQZ from 1.5-1.8kHz), and known losses from loss google sheet, we can infer some possible maximum and minimum arm powers using the no squeezing data.
The first attachment shows the same plot as above, but with the latest jitter noise measured by Elenna in 80808 We noticed this afternoon that there is a problem with the way these jitter noises are being added in quadrature by the noise budget, but we haven't fixed that yet. In this data set, we have 15.1dB of anti-squeezing and 5.1dB of squeezing from 1.5-1.8kHz, we can use the Aoki equations to solve for nonlinear gain of 14.6 and total efficiency eta for squeezing of 73%. Since the known readoutlosses are 7.3% and the known squeezer injection losses are 8.8%, this gives us a minimum readout efficency of (eta/(1-known injection loss) = 79% and a maximum of 1-known readout loss = 91.2%. Using the level of noise between 1.5-1.8kHz with no squeezing (and an estimate of the non quantum noise) we can use these max and min readout efficencies to find min and max circulating powers in the arms.
These arm power limits will be impacted by our estimate of the non-quantum noise, the homodyne angle, and the SRC detuning. With 0 SRC detuning, and a homodyne angle of 7 degrees, this resutls in a range of arm powers of 324-375kW. the estimate of non-quantum noise is the most important of these factors, while SRC detunings large engouh to change these estimates significantly seem outside the range that is allowed by other squeezing mesurements.
I've run the comparison of the model to different squeezing configurations for the low and high range and the nominal parameters (0 SRC, 7 degrees homodyne angle). Frequency independent squeezing and both types of mid squeezing are sensitive to the arm power from 50-100Hz, this comparison shows that the low end of the arm power range seems to have slightly too little arm power and the high range slightly too much. However these frequencies are also sensitive to homodyne angle and SRC detuning.
When Ibrahim relocked the IFO, the fast shutter guardian was stuck in the state "Check Shutter"
This was because it apparently got hung up getting the data using cdsutils getdata.
The first two attachments show a time when the shutter triggered and shows up in the HAM6 GS13s, and the guardian passes. The next time shows our most recent high power lockloss, where the shutter also triggered and shows up at a similar level in the GS13s, but the guardian doesn't move on.
The guardian log screenshot shows both of these times, it seems that it was still waiting for data and so the test neither passed nor failed.
To get around this and go to observing, we manualed to HIGH_ARM_POWER.
Vicky points out that TJ has sovled this problem for other guardians using his timeout utils, this guardian may need that added.
Another thing to do is make things so that we would notice that the test hasn't passed before we power up.
I looked at nds1's logs for this data request, the request appears to come it at the time it had timed out on h1guardian1.
From Shela's guardlog (paraphrasing somewhat):
2024-10-18_19:36:22Z timer
which is
2024-10-18T12:36:22 in "log PDT" format
NDS1 logs show (h1guardian1 is 10.101.0.249):
2024-10-18T12:36:27-07:00 h1daqnds1.cds.ligo-wa.caltech.edu daqd[1267959]: [Fri Oct 18 12:36:27 2024] connection on port 38943 from 10.101.0.249; fd=75
2024-10-18T12:36:27-07:00 h1daqnds1.cds.ligo-wa.caltech.edu daqd[1267959]: [Fri Oct 18 12:36:27 2024] ->42: version
2024-10-18T12:36:27-07:00 h1daqnds1.cds.ligo-wa.caltech.edu daqd[1267959]: [Fri Oct 18 12:36:27 2024] ->42: revision
2024-10-18T12:36:27-07:00 h1daqnds1.cds.ligo-wa.caltech.edu daqd[1267959]: [Fri Oct 18 12:36:27 2024] ->42: status channels 3 {"H1:ISI-HAM6_BLND_GS13Z_IN1_DQ"}
2024-10-18T12:36:27-07:00 h1daqnds1.cds.ligo-wa.caltech.edu daqd[1267959]: [Fri Oct 18 12:36:27 2024] ->42: status channels 3 {"H1:SYS-MOTION_C_SHUTTER_G_TRIGGER_VOLTS"}
So on first look it appears nds1 didn't get the request until after the 5 second timeout had expired.
Here is the request line (line 53) of isc/h1/guardian/LOCKLOSS_SHUTTER_CHECK.py
gs13data = cdu.getdata(['H1:ISI-HAM6_BLND_GS13Z_IN1_DQ','H1:SYS-MOTION_C_SHUTTER_G_TRIGGER_VOLTS'],12,self.timenow-10)
This request is for 12 seconds of data with a start time 10 seconds in the past, meaning it cannot complete until +2 seconds have elapsed. Does this mean the 5 second timeout is effectively a 3 second timeout?
PSL caused Lockloss as ASC and IMC lost lock within 7ms of one another.
Sheila, Vicky, remote inputs from Jenne and Jason
We noticed that more of our low noise locklosses have been tagged with Camilla's IMC tag (which we think is related to the PSL glitches) since the laser current was raised (8 out of 15 locklosses were tagged 10/15-10/18), compared to the time since the IMC servo gain redistribution revert (10/10 to 10/15 1 lockloss out of 12 was tagged). (Having this IMC tag made it very easy to do this check).
After emailing with the PSL team, we decided to turn the current back down to 2.133A at the next lockloss. Vicky's screenshot shows us slowly lowering the current at 20:05 UTC.
Fri Oct 18 10:16:11 2024 INFO: Fill completed in 16min 6secs
From 07:32 - 07:37 Fri 18oct2024 PDT we had another glitch on the EY CNS-II GPS 1PPS as read by the timing system's EY comparator. This had happened twice recently, 05:50-06:40 Mon 30sep2024 (50mins) and a pair the next day 07:14 (3mins) 07:42 (8mins) Tue 01oct2024.
attached are details of today's glitch, and a month trend showing all three.
On a related note, the EX CNS-II broke on the 8th Oct with a bad power supply (a wall wart). It might be prudent to schedule a replacement of EY's power supply next Tuesday.
It started again at 10:02. This time it is switching between -2200ns and -800ns
Follow up: EY CNS-II started glitching again around 10am Friday, it continued to do this for about an hour, then went good again. No further glitches since then over the past 23 hours.
R. Short, J. Oberling
After the NPRO control box was swapped last week and glitches persisted (see alog80566), we decided it would be best to switch back to the original so that signal readbacks for the NPRO would be accurate. I started by bringing down the PSL in a controlled manner (ISS, FSS, PMC, AMPs, NPRO), then went out to the LVEA PSL racks and swapped control box S/N S2200008 for S2200009. Since this is the control box that had previously been in service with this NPRO laser head, no adjustments were needed to set the temperature and current to be correct. I returned to the control room and brought the whole system back up without issue.
Once it was time to lock the FSS, similar to what was needed with the last control box swap, I manually moved the NPRO temperature down about 0.4K (from 0.19 to -0.23) and locked the RefCav so that the SQZ laser would be happy. I then updated the FSS search parameters and accepted them in SDF (see screenshot).
As I noted in the 10-day trends yesterday (see alog80665), PMC REFL has risen over the past week, so I attempted a remote alignment tweak using the picomotors before the PMC. In the end, I wasn't able to make much of any improvement to the PMC alignment. I also attempted a remote alignment tweak for the RefCav, and here I was able to get a slight improvement from 0.80V to 0.81V on the TPD.
Looking for more things to try to impact the laser glitching and hopefully bring down the amount of PMC reflected power, Jason and I returned to the PSL racks to try adjusting the NPRO pump current. We ultimately raised the current from 2.12A to 2.19A, increasing the NPRO output power from 1.82W to 1.91W according to the PD in front of the laser, but not having much of an impact on the PMC. We also tried slightly altering pump diode currents in the amplifiers, but we didn't see any improvement, so these remain as they were.
I concluded our PSL activities today with a rotation stage calibration following the steps in alog79596. The measurement file, new calibration fit, and screenshot of accepting SDFs are attached.
W (Max power in) | D | B (Min power angle) | C (Min power in) | |
Old Values | 94.642 | 1.990 | -24.794 | 0.000 |
New Values | 91.236 | 1.990 | -24.797 | 0.000 |
Original NPRO injection current was 2.133 A, and the new injection current is 2.193 A.