Unknown Lockloss at 00:52 UTC
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1378342360
Lockloss select plots attached.
Seismic System didn't have much motion.
I'll be looking more into this lockloss.
Lance, Genevieve, Robert
Recently, we shut down specific components of the HVAC system in order to further understand the loss of about 10 Mpc to the HVAC system (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72308 ). We noted that shutdown of the EX water pump had shown that the 52 Hz DARM peak is produced by the chilled water pump at EX. Based on coupling studies during commissioning time yesterday, the coupling of the water pump can be predicted from shaking injections in the area around the EX cryo-baffle, supporting the hypothesis that the water pump couples at the undamped cryo-baffle (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72769 ). Here we report on other results of the shutdown tests that we have been able to do so far.
CS Fans SF1, 2, 3, 4, 5, and 6 cost roughly 6 Mpc – coupling via input jitter noise and unknown coupling.
Figure 1 shows that the range increased by about 6Mpc when only the CS turbines were shut down; no chillers or chilled water pumps were shut down. Figure 2, a comparison of DARM spectra before, during, and after the fan-only shutdown, shows that there were two major differences. First, a decrease in peaks associated with input jitter noise, particularly the 120 Hz peak. Second, a broad band reduction in noise between about 20 and 80 Hz. This is not consistent with input jitter noise and represents an unknown noise source that we haven’t found yet.
There is a third difference that could be coincidence. The 9.8 Hz ITM bounce modes are higher in the before and after of Figure 2. I was tempted to wonder if the broad band noise was upconversion from the 9.8 Hz peak. We also have harmonics of roughly 10 Hz in the spectrum every so often. I compared BLRMS of 8.5-10 Hz to BLRMS of 39-50 Hz but didn’t see any obvious correlation. But Im not sure this eliminates the possibility.
120 Hz peak in DARM due to periscope resonance matching new 120 Hz peak from HVAC, possibly due to a new leak in LVEA ducts.
Figure 3 shows that the 120 Hz peak in DARM went away when only SF1, 2, 3 and 4 were shut down. It also shows that the HVAC produces a broad peak between 115 and 120 Hz. I looked back and the 120 Hz vibration peak from the HVAC appears to have started during HVAC work at the end of May, beginning of June. There was a period when flows were increased to a high level for a short time that might have pushed apart a duct connection that is now whistleing at 120 Hz. I think it would be worth checking for a leak in the ducts associated with SF1,2,3 and 4.
In addition to fixing a potential duct leak, we could mitigate the peak in DARM by moving the PSL periscope peak so that it doesn’t overlap with the HVAC peak. In the past I have moved PSL periscope resonances for similar reasons by attaching small weights.
EY HVAC does not contribute significantly to DARM noise
Figure 4 shows that on/off/on/off/on/off/on series of EY fan, chiller and water pump shutdowns does not seem to correlate with range.
This data is the analysis of the 2023-Aug-18 data originally summarized in LHO:72331.
TITLE: 09/09 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 44Mpc
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 19mph Gusts, 15mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY:
Inherited a Locked IFO that has been locked for 16:29 hours.
Wind is a little elevated.
TITLE: 09/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:
- Started out with a 6.0 EQ from Indonesia, but we were able to ride it out
- EX saturations @ 15:25
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 17:22 | EPO | Tours | LExC/CR/Overpass | N | Second Saturday tours! | 21:52 |
H1 is still going strong, currently locked for 12 hours. Ground motion has settled since this morning's earthquake. Second Saturday tours are ongoing.
Sat Sep 09 10:08:35 2023 INFO: Fill completed in 8min 31secs
TITLE: 09/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 3mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.10 μm/s
QUICK SUMMARY:
- H1 is currently locked, just hit the 8 hour mark
- FMCS channel still appear to be down on alert handler
- Literally as I was typing that seismic motion has calmed down since yesterday, we just got an alert for a 6.0 EQ from Indonesia en route...looks like I jinxed it
TITLE: 09/09 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 144Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
00:27 UTC Another Incoming 5.6M Earthquake from Indonesia.
Initial Alignment @ 1:23 UTC
Locking Started at @ 1:54 UTC
5.3M Eathquake from El Salvador 3:30UTC - Survived!
4 more 4.5-4.8M earthquakes roll through from Costa Rica , Panama, and Indonesia.
Lockloss 5:46 UTC https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72772
Back to NOMINAL_LOW_NOISE@6:53 UTC
Back to Observing @ 7:06 UTC
Unknown Lockloss https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1378273614
Plots look like there was possibly some movement of ASC-C soft before the actual lockloss.
While I did have two 4.5M Earthquakes enroute to pass through at the time, the ground motion was fairly low right as the lockloss happened. It's been quite a night of earthquakes so I had all the ground motion plots up and have been watching them like a hawk.
The PI plots look wild to me with the plateau on PImode 28 and 29 , so I took a pic of the log on SUS_PI .
There may be a smoking gun in there somewhere, but the PI's seem within what I recall being healthy bounds.
So cause is still unknown.
After a back to back set of earthquakes and an initial alignment.
H1 has made it back to NOMINAL_LOW_NOISE @ 02:58UTC.
There were SDF Diffs I had to accept to get into Observing @03:01 UTC.
SUS-FC2_M1_OPTICALALIGN_P_OFFSET
SUS-FC2_M1_OPTICALALIGN_Y_OFFSET
Tagging SUS team to documen tthe SDF Diffs.
Tagging ISC and SQZ teams:
(1) Why are the FC2 alignment offsets monitored by the SDF system?
(2) Why would we have expected a change in the FC2 alignment? Is this concurrent and/or a result of other commissioning / optimizing?
1) FC2 alignment offsets in principle don't have to be monitored by the SDF system anymore, since we have commissioned ASC. While commissioning the squeezer, at some point it was helpful to do it this way before we set up ASC, but I don't think it makes a big difference now that filter cavity ASC is running. Sheila has already un-monitored several other alignment offsets in ZMs recently, so it should be fine to do that here as well.
2) FC2 alignment changed to help the filter cavity lock on a TEM00 mode after the string of earthquakes on Friday. There was significant pitch misalignment, so Naoki manually helped the FC green lock catch by aligning the FC2 (mostly pitch) slider. If FC does not catch green lock (similar to if ALS does not catch green lock due to misalignment) the system won't be able to go to FDS. So in this case, it can be nice to help it a bit by bumping FC2 P/Y sliders to get the green spot locking on FC green transmission camera (nuc33, bottom left).
Shaker injections support the hypothesis that the 52 Hz peak from the HVAC chilled water pump is coupling through the cryopump manifold baffle at EX. The attached spectrum shows a 50 Hz sinusoidal injection of a shaker in red, shaking the vacuum enclosure around the crypump manifold baffle, which also shows up in the PEM channels. Using this injection to calculate the vibrational coupling factor to DARM, and then multiplying by the magnitude of the 52 Hz HVAC peak in the PEM channels, results in an expected contribution to DARM within about 75% magnitude of the observed 52 Hz peak in DARM. We consider power coupling functions good to a factor of 2, so 75% is reasonable.
Late entry:
On Tuesday, the Kobelco was started to use the dry air to purge one of the turbo headers located in the LVEA. After starting the booster pump, a very small leak was found while performing the pre-starting inspection of the compressor, two drops of coolant were found on the catch pan inside the enclosure, the leak was traced to the gasket on the joint for the flanges of the "intercooler shell" and the "inlet adapter" (see on "Diagram" items 1, 3, and 4) unfortunately no torque specifications for the bolts holding both pieces together were found, so we did not wrench on it. A vendor was contacted and they are send a technician to diagnose/fix the leak on Monday.
The Kobelco ran for about 2.5 hours, the leak was monitored during such time, no noticeable increase on the leak was noted, but remained present. The drying skid was turned on and the RH was measured, -41.1 degrees C for one tower, the other tower read -42.3 degrees C, RH was monitored during the period of operation and no changes were noted.
TITLE: 09/08 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
SEI_ENV state: LARGE_EQ
Wind: 11mph Gusts, 8mph 5min avg
Primary useism: 5.51 μm/s
Secondary useism: 0.27 μm/s
QUICK SUMMARY:
Picket Fence and Seismic System are all ringing up still from a 6.8M earthquake out of Morocco. Seems Like I'll be waiting for the uSeism to fall before attempting to lock.
TITLE: 09/08 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Down
INCOMING OPERATOR: Tony
SHIFT SUMMARY:
- EX saturation @ 15:35
- 16:00 - EQ mode activated
- Lockloss @ 16:10 - caused by a Mexico EQ, few sizable aftershocks followed (5.8), holding in READY until ground motion settles
- PEM commissioning work went from 19:00 - 21:00 UTC
- Got back to NLN @ 19:13
- Relocking was autonomous until a 6.8 EQ from Morocco decided to show its ugly face...holding in READY to ride out the ground motion
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 16:12 | PEM | Robert | EX | Y | Viewport cover changes | 17:14 |
| 16:16 | FAC | Kim | H2 | N | Tech clean | 16:46 |
| 16:35 | FAC | Tyler | EY | N | Check pump lines on AHU | 17:12 |
| 17:12 | FAC | Tyler | CS | N | Check drip pan | 17:15 |
| 17:29 | SQZ | Vicky/Louis | SQZ table | Y (LOCAL) | SQZ alignment | 18:32 |
| 19:34 | PEM | Robert | EX | N | PEM commissioning | 20:45 |
| 19:50 | FMCS | Bubba | EX | N | Check chiller alarm | 20:06 |
| 19:58 | FAC | Tyler | EY | N | Check chiller alarm | 20:08 |
| 20:18 | FAC | Bubba | LVEA | N | Test HVAC fans | 20:26 |
Lockloss @ 21:41, cause unknown.
To archive our lock-loss-alert changes using the userapps subversion repository, I have written a script which automatically accepts any pending contact changes, and commits the resulting burt snap file to the userapps repository. The commit string explicitly notes that this is a robot commit, and gives the date-time of the commit.
The script runs periodically as a cronjob.
For the initial roll-out, the cronjob runs on opslogin0 every 15 minutes as david.barker, and uses my current Kerberos ticket for authentication with the subversion repository server.
This service will eventually move to a CDS server and use a robot authentication ticket.
Note that currently the contact channels are indexed by ID. The MEDM needs to be consulted to map an ID to the contact's name.
Louis, Vicky
In 72525, on SQZT7 Sheila and I adjusted the beamsplitter angle between the OPO_IR_TRANS PD & camera, which changed the beam-splitter ratio between refl / trans slightly. Today Louis and I re-measured and updated the OPO_IR_PD calibration. If we want to try a SQZ single-bounce measurement through the AS port, it'll be important to have an accurate calibration of the squeezer's injected power at the beam diverter. I think this calibration update is likely good to within ~2% (and very likely good to within ~3.5%)**.
With ~10mW into the seed fiber, using the Ophir PD (calibrated 10/18, at least the filter is part of S/N 889882), at 11:05AM PT we measured 52.7uW on the OPO_IR_TRANS PD, and then at 11:08am PT, we measured like OPO_IR_TRANS = 99-100uW going into the beam-splitter. While the epics calibration was fine, the slowcontrols calibration was off, so we accordingly updated the PD responsivity (0.23 to 0.235) and the PD splitter ratio (59% to 53.5%); SDF screenshot attached.
** This is assuming we trust the Ophir PD calibration into watts, but this PD was last calibrated 2018, and it wanted to be re-calibrated in 2020, but was not. We first measured with a Thorlabs PD, but that PD was inaccurate and reported the uW high by >10%. We are assuming the Thorlabs PD is inaccurate b/c the Ophir reading was closer to the former PD calibrations in slowcontrols/epics, so we are deciding to trust that one.
While FMCS work is ongoing, I have bypassed the cell-phone alarms for the channels shown below:
Bypass will expire:
Fri 08 Sep 2023 03:06:06 PM PDT
For channel(s):
H0:FMC-CS_CY_H2O_PUMPSTAT
H0:FMC-CS_CY_H2O_SUP_DEGF
H0:FMC-CS_FIRE_PUMP_1
H0:FMC-CS_FIRE_PUMP_2
H0:FMC-CS_WS_RO_ALARM
H0:FMC-EX_CY_H2O_PUMPSTAT
H0:FMC-EX_CY_H2O_SUP_DEGF
H0:FMC-EY_CY_H2O_PUMPSTAT
H0:FMC-EY_CY_H2O_SUP_DEGF
All the FMCS EPICS channels have been static since around 11:00 this morning. The attached plot shows a representative channel from CS and each out-building showing they stopped updating between 10:47 and 11:06. Interestingly most of the EPICS channels did not go invalid, but some did. This explains the mixture of green/white values on the FMCS MEDMs seen this afternoon. Jonathan and Patrick restarted the FMCS IOC at 15:01, at which point all the EPICS channels went to VAL=0, SEVR=Invalid.
At the time of writing, the corner station chiller-yard and wood-shop FMCS channels continue to be good. Therefore I have un-bypassed the critical fire-pump alarms for the weekend.
The current cell-phone alarm bypass list is now
Bypass will expire:
Mon 11 Sep 2023 03:16:29 PM PDT
For channel(s):
H0:FMC-CS_CY_H2O_PUMPSTAT
H0:FMC-CS_CY_H2O_SUP_DEGF
H0:FMC-CS_WS_RO_ALARM
H0:FMC-EX_CY_H2O_PUMPSTAT
H0:FMC-EX_CY_H2O_SUP_DEGF
H0:FMC-EY_CY_H2O_PUMPSTAT
H0:FMC-EY_CY_H2O_SUP_DEGF
Elenna, Sheila
We got data today to rerun our noise budget with the current noise (150Mpc). We got quiet time with no squeezing for 10 minutes starting at 1375723813, with no large glitches. We ran excitations for LSC, laser noise, and ASC. We had quiet time with squeezing injected from the previous night of observing, I choose 1375695779 as a time with high range and no large glitches. This is commit 50358cda
Elenna, Sheila
We ran the noise budget code for this no squeezing time.
This is all commited as 0f9ffe0e
Sheila, Vicky - we have re-run the noise budget for following times:
Noise budget with squeezing. Changes here: using GDS instead of CAL-DELTAL, closer thermalized FDS time to no-sqz, using updated IFO gwinc parameters related to quantum noise calculation.
(Edit: was a glitch in the old time; updated to an FDS time without glitches. All plots updated.)
PDT: 2023-08-10 08:45:00.000000 PDT
UTC: 2023-08-10 15:45:00.000000 UTC
GPS: 1375717518.000000
PDT: 2023-08-10 09:35:52.000000 PDT
UTC: 2023-08-10 16:35:52.000000 UTC
GPS: 1375720570.000000
Noise budget with no squeezing. Same time as above, now calculates using gwinc quantum noise calculation instead of semiclassical calculation used previously.
PDT: 2023-08-10 10:18:11.000000 PDT
UTC: 2023-08-10 17:18:11.000000 UTC
GPS: 1375723109.000000
Both sqz & no-sqz noise budgets now use the correlated quantum noise calculation from gwinc, instead of semiclassical calculations for SN & QRPN. The gwinc budget parameters related to quantum noise calculation are consistent with the recent sqz data set (8/2, alog 72565), with readout losses evenly split between IFO output losses that influence optical gain (20%) and SQZ injection losses (20%), parameters in plot title here. This is high on SQZ injection losses, and slightly conservative on IFO output losses. This updated FDS time is thermalized and closer to the No-SQZ time; the time used previously was several hours earlier near the start of lock, w/ ifo not yet thermalized.
Unlike before, both budgets now show GDS-CALIB STRAIN, which on 8/10 was more accurately calibrated (see Louis's alog on Aug 8, LHO:72075, comparing CAL-DELTAL and GDS vs. PCAL sweep, and his record from 72531). CAL-DELTAL was previously overestimating range due to calibration inaccuracies. We got GDS-CALIB_STRAIN data from nds servers, and at first weren't able to get input jitter data from nds, due to the sampling rate change of IMC-WFS channels from 2k to 16k, 71242. Jonathan H. helped us fix this issue, so we can now pull GDS data and input jitter data from nds.ligo-wa.caltech.edu:31200 -- thank you Jonathan!! With this, the input jitter sub-budget is kind of interesting, looks to be mostly IMC-WFS in YAW.
A quick thought on discrepancy between expected and measured DARM between below several hundred Hz-- I don't know if this could be related to the recent update to gwinc CTN parameters (high/low index loss angles), related to quantum noise, or mystery noise. The recent gwinc CTN update seemed to have dropped the calculated CTN level slightly (maybe 10-15% or so). In April 2023, Kevin helped update CTN parameters LHO:68499 to reconcile H1 budget with the official gwinc parameters, while Evan made a correlated noise measurement 68482 where noise in the bucket seems more consistent with the older CTN estimate from gwinc (or very slightly higher). Another idea is that it could be related to quantum noise, such as SRCL detuning or sqz angle which could've changed since the sqz dataset, as quantum noise can also affect the noise in this region.
All pushed as git commit 28cf2664.
Edit: All pushed again as git commit 33ffd60b.
Added noise budgets with squeezing for HVAC off time on August 17 from alogs 72308, 72297.
When comparing this HVAC off time on Aug 17 with the noise budget from above on Aug 10, it's interesting to note the broadband difference in input jitter (Aug 10 vs Aug 17, HVAC off). Between these times, worth noting that I think there were several additional improvements (like LSC FF or SUS-related) as well.
Edit: updated 8/10 input jitter budget to the less glitchy noise budget time.
Much of the gap between expected DARM (black traces) and measured DARM (red traces) in the noise budget looks compatible with elevating the CTN trace. Budget plots with 100 Hz CTN @ 1.45e-20 m/rtHz are attached below for the no-HVAC times. This is almost 30% higher than the new gwinc nominal CTN at 100 Hz (i.e., 1.128e-20 m/rtHz --> 1.45e-20 m/rtHz). Compared to the old gwinc estimate of 1.3e-20, this is ~11% higher. Quantum noise calculation unchanged here.
This CTN level is similar to the 30% of excess correlated noise that Evan H. observed in April 2023, see LHO:68482. His cross-correlation measurement sees ~30% excess correlated noise around 100 Hz after subtracting input jitter noise, where that "30%" is using the newer gwinc CTN estimate of 1.128e-20 m/rtHz @ 100 Hz. This elevated correlated noise, if attributed to CTN, corresponds to CTN @ 100 Hz of about 1.3*1.128 = 1.46e-20 m/rtHz. See this git merge request for the gwinc CTN update ; this update lowered the expected CTN at 100 Hz by ~15%, from 1.3e-20 (old) to 1.1e-20 m/rtHz (new), based on updated MIT measurements.
For reference, I have plotted these various CTN levels as dotted traces in the thermal sub-budget.
To elevate CTN levels by 30% in the budget code, I scaled both high+low index loss angles by a factor of 1.8, specifically Philhighn 3.89e-4 --> 7e-4 ; Phillown 2.3e-5 --> 4.14e-5. It seems like much higher than this level ~1.45e-20 might be difficult to reconcile with the full budget.
Noteworthy w.r.t. squeezing: from the laser noise sub-budget, laser frequency noise looks within 33% of squeezed shot noise with ~3.7dB of squeezing. By contrast, the L1 noise budget from Aug 2023 (LLO:66532) shows laser noise at the ~20% level of squeezed shot noise with 5.3 dB of squeezing -- i.e. a lower laser noise floor past shot noise.
The following plots can be found in /ligo/gitcommon/NoiseBudget/aligoNB/out/H1/lho_all_noisebudgets_081723_noHVAC_elevatedCTN, and not yet commited to the git repo.
Plots with higher CTN are attached here for the SQZ / no-SQZ proper noise budget times from 8/10, when injections were run.
Comparing the sqz vs. no-sqz budgets suggests there might be more to understand here, to tease apart the contributions from coating thermal noise (CTN) vs. quantum noise in the bucket. In particular, something disturbing that stands out, is that I imagined that if elevated CTN is the physical effect we're missing, it would reconcile both NBs with and without squeezing. However, there is still some discrepancy in the un-squeezed budget, which was not resolved by CTN, and seems to have a consistent shape. I'm wondering if this is related to the IFO configuration as it affects the quantum noise without squeezing. I think this could result from a non-zero but small SRCL detuning since it looks like elevated noise, with a clear shape, that increases below the DARM pole. Simply elevating CTN to match the no-sqz budget would put us in conflict with squeezed darm, so I don't think it makes sense to elevate CTN further. The budget currently has 0 SRCL detuning as it "seems small-ish", but this parameter is somewhat unconstrained in the quantum noise models.
In models, the readout angle is upper-bounded by Sheila's contrast defect measurement, though in principle it could probably be anything lower than that too, which could be worth exploring. It might be helpful to have an external measurement of the thermalized physical SRCL detuning, or in the models allowing the SRCL detunings to vary, to explore how it fits or is constrained by the fuller noise budget picture.
Plots with squeezing can be found in /ligo/gitcommon/NoiseBudget/aligoNB/out/H1/lho_all_noisebudgets. No squeezing plots are in /ligo/gitcommon/NoiseBudget/aligoNB/out/H1/lho_darm_nosqz_noisebudget.
I pushed to git commit 70ca191c without elevated CTN and the associated extra traces. The relevant parameters are left commented out at the bottom of the QuantumParams file, and relevant code to plot the extra traces is commented out in the lho_all_noisebudgets script.