TITLE: 11/13 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 16mph Gusts, 11mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.65 μm/s
QUICK SUMMARY: Looks like H1 made it up to low noise two times in the last 3 hours, but lost lock immediately. useism is still quite high, and some wind overnight didn't help. The IMC is now struggling to stay locked after a few failed DRMI attempts, but a successful PRMI. Starting initial alignment now.
TITLE: 11/13 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
Slow start to locking at the beginning of the shift, but once PSL work was done, ran an alignment and returned to locking (had one lockloss due to PSL/IMC), then made it to NLN/Observing and had a "normal lockloss" (i.e. NOT PSL/IMC).
Probably old news, but it feels like after locklosses DRMI/PRMI have been pretty misaligned. Sometimes fixed by CHECK MICH FRINGES/PRMI, but sometimes a full INITIAL ALIGNMENT is required. (Running 2nd Initial Alignnment of the shift before the end of the shift.)
Microseism is still high but winds calmed during this shift.
LOG:
E.Capote, C.Gray
With the recent lockloss, Elenna and I went out to do the LVEA Sweep, which also included unplugging the Freq Noise Injection cable they plugged in yesterday.
Long Maintenance/Preventative-Maintenance Day Today
Once Jason & RyanS finished their PSL work on the floor (and some observing of the IMC for glitches in the Control Room), went for locking at about 515pm PST---with an INITIAL ALIGNMENT which completed with no major hitches. For locking, we continue to get Locklosses before/at LOCKING ARMS GREEN (so held at READY for a bit....but we had to wait a few minutes for the IMC to lock). Once we got past this, went for our first lock of the night.
[µseism is still very high, but winds dropped from 30mph down to 10mph]
Lock #1: LOCKLOSS for H1 at TRANSITION FROM ETMX[557] (which is noteworthy because Elenna was wanting to see if the lowered T-Ramp (alog81224) at LOWNOISE ESD ETMX [558] and we didn't get to see this. Vicky looked at this lockloss and sadly this LOCKLOSS was one of our infamous PSL/IMC Glitches.
Lock #2: After the lockloss above, DRMI looked pretty bad, so I took it to PRMI (but it also needed CHECK MICH FRINGES). After that, H1 went all the way to NLN automatically. The Violins were high (~500Hz peak was at around 1e-15), so we had to stay at OMC WHITENING for 20min before going to NLN. Then there were a list of SDFs which were all ACCEPTED (see below and attached). H1 to OBSERVING at 0340utc with a range hovering just under 160Mpc.
Maintenance Day Notes:
SDF Diffs That Were ACCEPTED: (see attached)
Lockloss at 0442utc after about 1hr of Observing.
BUT---Vicky was saying this was a "normal" (i.e. not PSL/IMC glitch lockloss).
Lockloss #1 @ 1415499897 - likely PSL/IMC.
Standard PSL trends show FSS glitch, IMC, and IFO lockloss all at the same time (< 16kHz). Trends including SQZ not illuminating; see SQZ FSS glitch same time as PSL glitch (same time as lockloss), but not more info.
Lockloss #2 @ 1415508172 - NOT PSL/IMC! Looks like a "standard" (?) lockloss.
Standard PSL trends show that IFO and AS port see the lockloss ~200ms before the IMC loses lock. This was the typical lockloss before IMC/PSL issues. SQZ trends are normal; SQZ TTFSS and signals lose it when PSL FSS loses lock (again, ~200ms after arms see the lockloss), as expected.
At 19:27:26 PST a PT132_MOD2 glitch caused a single-sensor VACSTAT glitch. The new code requires two chambers to glitch before alarming the VAC team, so in this case no cell phone alarms were sent.
The glitch lasted 4 seconds, and as before it was DeltaP which tripped, exceeded its trip level of 1.0e-08Torr (see attached).
I restarted vacstat_ioc.service on cdsioc0 to reset the system.
TITLE: 11/13 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 26mph Gusts, 19mph 3min avg
Primary useism: 0.67 μm/s
Secondary useism: 0.77 μm/s
QUICK SUMMARY:
Long Maintenance Day, high microseism, high winds, M5.0 Mexico earthquake.....but the IMC was just locked in the last few minutes!
TITLE: 11/13 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
H1 was held in IDLE all day while we had a long maintenance day with a focus on the PSL's TTFSS repair.
The PSL Crew is still in the PSL room, re-installing the TTFSS.
Secondary microseism is still elevated wind is still elevated as well.
Beckhoff updates were done today at 4PM as well.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
21:41 | SAF | Laser | LVEA | YES | LVEA is laser HAZARD | 08:21 |
15:32 | FAC | Karen | Optics & VAC Prep Labs | N | Technical cleaninng | 15:42 |
15:42 | FAC | Karen | EY | n | Technical Cleaning. | 17:01 |
15:55 | FAC | Nelli | HAM Shaq | N | Technical Cleaning | 16:41 |
15:56 | FAC | Kim | EX | N | Technical Cleaning | 17:21 |
16:30 | SEI | Neil, Jim, Fil | LVEA Beirgarden | yes | Installing GeoPhone Seimometer | 19:23 |
16:46 | 3IFO | Tyler | LVEA | Yes | 3 IFO Checks | 17:22 |
16:57 | PSL | Jason & Ryan S | PSL Room | Yes | Working on PSL, PMC, TTFSS | 18:41 |
17:11 | SUS | Rahul | EY & EX | N | SUS Measurements | 18:39 |
17:13 | CDS | Richard | CER | N | Maximizing the window for Maintenance | 17:16 |
17:15 | N2 | NorCo | Y arm | N | N2 fill, another Nor Cal Truck also went down the Y arm | 19:15 |
17:22 | FAC | Tyler | Mid Stations | N | 3 IFO checks | 19:22 |
17:39 | pSL | Daniel | PSL racks | Yes | Removing power from old PSL rack equipmnt | 19:39 |
17:45 | VAC | Travis & Janos | EY & MX | N | Checking vac pump status in Recieving areas | 19:39 |
17:51 | FAC | Kim & Karen | LVEA | YES | Technical Cleaning Karen out early | 19:04 |
18:30 | SEI | Neil D. | LVEA | Yes | Taping a GeoPhone to the Floor. | 18:50 |
18:57 | SEI | Neil | CER | N | Turning on a switch in the CER | 19:14 |
19:05 | FAC | Kim | High bay | n | Tech clean | 20:03 |
20:08 | VAC | Janos | EY, MX | N | Vac Hardware check. | 22:08 |
20:58 | PSL | Daniel | LVEA PSL Racks | Yes | Checking on the FSS | 21:51 |
21:14 | FAC | Erik Tyler | Mid Y | N | Handling the Air , Tyler back early | 22:36 |
21:41 | ISC | Christina | EY, MX | n | Inventory | 22:16 |
21:47 | PSL | Ryan S & Jason | PSL | Yes | Replacing the TTFSS | 01:48 |
22:01 | VAC | Travis | MX | N | Vac systems check | 22:20 |
22:07 | EE | Fil | CER | N | Turning on High Voltage | 22:17 |
23:22 | VAC | Janos | Mid X | N | Measuring Vac system equipment. | 23:34 |
23:47 | SEI | Jim | CER | N | checking cables | 23:52 |
23:54 | CDS | Erik | CER | N | Getting a laptop | 00:00 |
Famis 26336
Just over 6 days ago there was an increase in H0:VAC-EX_FAN1_570_2_ACC_INCHSEC
Both H0:VAC-MR_FAN1_170_1 & 2 ACC_INCHSEC saw an increase in noise in the last 3 days.
TITLE: 11/12 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 33mph Gusts, 26mph 3min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.85 μm/s
QUICK SUMMARY:
17:11 UTC Test T524646 EXTTRIG: SNEWS alert actve
Maintenance agenda:
Saw 1 PSL glitch ~50 minutes after Jason/RyanS (removed the TTFSS box and calibrated the H1:PSL-PWR_HPL_DC_OUT_DQ channel). Otherwise was quiet ~ 3 hours.
Trends here. Glitch visible on mixer with peak-peak +/-0.1, similar to seen yesterday with FSS autolocker off. These are not the biggest mixer glitches we have seen, but are the large ones (like glitch 3, or end of glitch 2 yesterday). Jason notes the temperature feedback cable is still plugged in (though the FSS box has been removed).
We expect mixer output to be fuzzier (bigger RMS) when FSS is off, because FSS is not running to control the laser frequency noise w.r.t. refcav. So, should be seeing the free-running npro laser linewidth with FSS off, which is expected to be more than with FSS on (aka, no need to get thrown off by the mixer output fuzziness with FSS off).
The VEAs in Mid X and Mid Y have four temperature sensors located inside the exterior wall. These four sensors are averaged by the automation system for the purpose of temperature control. The location in the exterior wall means all four of these sensors are influenced by outdoor ambient air and consistently read higher or lower than the interior of the VEA, depending on the season. At both mid stations, one of these sensors will be relocated to the interior of the room. Then, two of the sensors in the exterior wall will be taken out of the averaging so that one interior sensor and one wall sensor will be used to figure the average space temperature. This will have an impact on temperature trends for the mid stations, and although these are not critical like the temperatures at the corner and end stations, anyone keeping an eye on the trends will notice a significant change.
Image 1: PSL top eurocrate
From left to right: ISS inner loop, old ISS second loop (depowered), reference cavity heater (depowered), and TTFSS field box.
Image 2: PSL bottom eurocrate
From left to right: injection locking field box (depowered), injection locking servo (depowered), monitor field box, PMC locking field box, and PMC locking servo.
Tue Nov 12 10:07:17 2024 INFO: Fill completed in 7min 13secs
Jordan confirmed a good fill curbside.
TITLE: 11/12 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 18mph Gusts, 14mph 3min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.78 μm/s
QUICK SUMMARY:
Good Maintenance Tuesday Morning Everyone.
H1 was in move spots when I walked in and shortly there after caught a lockloss.
So In_Lock_SUS_Charge Measurements will not be ran this week.
List of expected Maintenance Items :
After discussion with Tony we decided not to install his new PCAL Guardian nodes today, which means no DAQ restart today (my VACSTAT ini change is target-of-opportunity if a DAQ restart were to happen for other reasons)
We seem to be having lots of locklosses during the transition from ETMX/lownoise ESD ETMX guardian states. With Camilla's help, I looked through the lockloss tool to see if these are related to the IMC locklosses or not. "TRANSITION_FROM_ETMX" and "LOWNOISE_ESD_ETMX" are states 557 and 558 respectively.
For reference, transition from ETMX is the state where DARM control is shifted to IX and EY, and the ETMX bias voltage is ramped to our desired value. Then control is handed back over the EX. Then, in lownoise ESD, some low pass is engaged and the feedback to IX and EY is disengaged.
In total since the start of O4b (April 10), there have been 26 locklosses from "transition from ETMX"
From the start of O4b (April 10) to now, there have been 22 locklosses from "lownoise ESD ETMX"
Trending the microseismic channels, the microseism increases seems to correspond with the worsening of this lockloss rate for lownoise ESD ETMX. I think, when correcting for the IMC locklosses, the transition from ETMX lockloss rate is about the same. However, the lownoise ESD ETMX lockloss rate has increased significantly.
Here is a look at what these locklosses actually look like.
I trended the various ETMX, ETMY and ITMX suspension channels during these transitions. The first two attachments here show a side by side set of scopes, with the left showing a successful transition from Nov 10, and the right showing a failed transition from Nov 9. It appears that in both cases, the ITMX L3 drive has a ~1 Hz oscillation that grows in magnitude until the transition. Both the good and bad times show a separate oscillation right at the transition. This occurs right at the end of state 557, so the locklosses within 1 or 2 seconds of state 558 likely are resulting from whatever the of the steps in state 557 do. The second screenshot zooms in to highlight that the successful transition has a different oscillation that rings down, whereas the unsuccessful transition fails right where this second ring up occurs.
I grabbed a quick trend of the microseism, and it looks like the ground motion approximately doubled around Sept 20 (third screenshot). I grabbed a couple of recent successful and unsuccessful transitions since Sept 20, and they all show similar behavior. A successful transition from Sept 19 (fourth attachment) does not show the first 1 Hz ring up, just the second fast ring down after the transition.
I tried to look back at a time before the DAC change at ETMX, but I am having trouble getting any data that isn't minute trend. I will keep trying for earlier times, but this already indicates an instability in this transition that our long ramp times are not avoiding.
Thanks to some help from Erik and Jonathan, I was able to trend raw data from Nov 2023 (hint: use nds2). Around Nov 8 2023, the microseism was elevated, ~ 300 counts average on the H1:ISI-GND_STS_ITMY_Z_BLRMS_100M_300M channel, compared to ~400 counts average this weekend. The attached scope compares the transition in Nov 8 2023 (left) versus this weekend Nov 10 (right). One major difference here is that we now have a new DARM offloading scheme. A year ago, it appears that this transition also involves some instability causing some oscillation in ETMX L3, but the instability now creates a larger disturbance that rings down in both the ITMX L3 and ETMX L3 channels.
Final thoughts for today:
Sheila and I looked a look at these plots again. It appears that the 3 Hz oscillation has the potential to saturate L3, which might be causing the lockloss. The ringup repeatedly occurs within the first two seconds of the gain ramp, which is set to a 10 second ramp. We decided to shorten this ramp to 2 seconds. I created a new variable on line 5377 in the ISC guardian called "etmx_ramp" and set that to 2. The ramps from the EY/IX to EX L3 control are now set to this value, as well as the timer. If this is bad, this ramp variable can be changed back.
Sheila and I went out to the floor to plug in the cable to run frequency noise injections (AO2).
I switched us to REFL B (79037) only to run the injections. I first ran the frequency noise injection template. Then, I disenaged the 10 Hz pole and engaged the digital gain to run the ISS injections (low, medium, and high frequency). I was fiddling with the IMC WFS pitch injection template to get it to higher frequency for a jitter injection when we lost lock.
Next step is to at least use the intensity and frequency measurements to determine how the noises are coupling to DARM (would be nice to have jitter too, but we'll see).
Commit 464b5256 in aligoNB git repo.
These results show that above 3 kHz, some of the intensity noise couples through frequency noise. There is no measurable intensity noise coupling through frequency noise at mid or low frequency.
The frequency noise projection and coupling in this plot are divided by rt2 to account for the fact that frequency noise is measured on one sensor here, but usually is controlled on two REFL sensors. This may not be correct for high frequency (roughly above 4 kHz), where CARM is gain limited.
code lives in /ligo/gitcommon/NoiseBudget/simplepyNB/Laser_noise_projections.ipynb
Two of the 4 locklosses over the night have FSS_OSCILATION tags and two have the IMC tag. The lockloss tool and attached trend show more glitches in the H1:PSL-FSS_FAST_MON_OUT_DQ channel before these locklosses. Maybe from the changes yesterday: TTFSS changed and "PSL FSS: Common Gain increased from 13 to 15. Fast Gain increased from 5 to 6" 81235
NLN locklosses last night:
Increasing the gains should not have caused more oscillations. I'm writing an alog covering the entirety of our work yesterday right now, but we raised the gains due to finding a bad component in the in-service TTFSS (the PA-85 high voltage op-amp in the EOM path) whose replacement
removedreduced the observed zero crossings around 500kHz noted by Sheila et al here and corrected anomolies Marc observed in the EOM path TF (he has the data for this). We also correctly tuned the EOM notch to remove the 1.685MHz peak also seen in the linked alog. Raising the common gain put the FSS UGF back in its normal range (400kHz - 500kHz), and we raised the Fast gain by 1dB to compensate for a slight loss of gain in the PZT path caused by our moving the PZT notch from ~35.7kHz to ~32.7kHz. The TF and crossover measurement showed all should be well with the FSS after these fixes, so it's still unclear why these IMC locklosses continue.Edit: Upon closer inspection of the TF, the peakiness around 500kHz is still present but apparently less, yet still close to a zero crossing. We could lower the Common gain by 1dB (to 14dB) to better clear the zero crossing if we suspect this is causing instability.
At 17:18UTC had another lockloss where the FSS_FAST_MON and FSS_PC_MON grow before the lockloss, IMC caused. Plot, zoomed plot shows there isn't explicit glitches but the FSS_FAST_MON channel just gets more noisy.
Zoom out comparing Camilla's Lockloss 1 (tagged "FSS" + "IMC") vs. Lockloss 2 (tagged "FSS").
LL1 1415545894 sees lots of FSS glitches before FSS and IMC lose lock, then IFO loses lock. LL2 1415542310 sees noise related to FSS grow slowly before lockloss.
For lockloss 2, tagged "FSS" with a slow oscillation growing before lockloss, Daniel and I then compared the FSS and OMC_DCPD fast (64k) spectra before losing lock here (where H1:PSL-FSS_FAST_MON_OUT_DQ has been calibrated with zpk = ([], [10], [1.3e6]) to get Hz/rtHz.
After PSL swap (cyan and pink traces) - laser noise looks as Daniel expected with ~100 Hz/rtHz at 100 Hz. Before PSL swap, black trace looked lower than expected, unclear. From lho81210, LLO engages this zpk filter in their filter bank to get the output of H1:PSL-FSS_FAST_MON_OUT_DQ in Hz/rtHz ; LHO has it in the filter bank but does not engage it.
We see more broadband extra noise in FSS (not in OMC DCPD) before losing lock, but no big peaks / features otherwise.