TITLE: 12/19 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.30 μm/s
QUICK SUMMARY:
Detector Observing and Locked for 1.5 hours. Seems like everything has been going well.
TITLE: 12/19 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Average maintenance day that ran just a bit long and a straightforward locking process.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
22:41 | SQZ | Nutsinee | Optics Lab | Local | Ongoing | |
15:05 | cal | Louis | CR | N | testing new DARM loop config | 16:00 |
15:48 | FAC | Randy | Outside/ High bay | N | Driving Forklift | 16:08 |
15:59 | FAC | Karen | Optics Lab | n | Technical Cleaning | 16:14 |
16:32 | IOO | Ryan S | CR | - | Center IMC WFS | 16:47 |
16:00 | FAC | Kim | LVEA | N | Technical cleaning | 17:32 |
16:14 | FAC | Karen | LVEA | - | Technical cleaning | 17:33 |
16:04 | FAC | Mitchell, Randy | LVEA | - | HAM3 cleanroom | 17:39 |
16:19 | FAC | Cindi | FCES | - | Technical cleaning | 17:40 |
17:25 | IAS | Jason, Randy | LVEA | - | FARO craning | 17:44 |
17:35 | EE | Marc | EX | - | Checking power supplies | 17:57 |
17:49 | SAF | Camilla, Jason | LVEA | YES | Transition to HAZARD | 17:59 |
16:38 | VAC | Travis | LVEA | - | HAM5 turbopump line work | 18:15 |
17:59 | EE | Marc | CER | - | Checking power supplies | 18:17 |
18:17 | EE | Marc | EY | - | Checking power supplies | 18:43 |
18:29 | FAC | Randy | LVEA | YES | 18:50 | |
18:09 | OPS | Mitchell | EX, EY | - | FAMIS checks | 18:51 |
18:12 | FAC | Karen | EY | - | Technical cleaning | 18:58 |
18:46 | EE | Marc | LVEA | YES | Talk to Camilla | 19:06 |
18:13 | EE | Fil | LVEA | YES | Connect temp sensor near BSC2 | 19:07 |
18:11 | FAC | Tyler, FM | EY | - | Fire safety checks | 19:15 |
19:21 | IAS | Tyler | LVEA | YES | Grab parts for FARO | 19:27 |
15:58 | S&K elect | Ken | EX | - | 19:37 | |
16:43 | VAC | Janos | EX, MX | - | Hepta pump checks | 19:56 |
19:07 | EE | Marc, Fil | EY | - | Check TCS power supply | 19:57 |
19:17 | FAC | Chris | All Buildings | - | Safety equipment checks | 20:00 |
16:23 | VAC | Gerardo | LVEA | - | Check on HAM2 AIP | 20:01 |
18:41 | SQZ | Daniel | LVEA | YES | SQZ parts near HAM6 | 20:07 |
20:16 | SQZ | Sheila, Naoki | LVEA | YES | Turning sidebands back on | 20:18 |
16:48 | SQZ | Vicky | Remote | - | SQZ/OMC scans | 20:26 |
19:43 | SEI | Jim | EX | - | Recover HEPI pump station | 20:26 |
20:04 | EE | Fil, Marc | EY | - | Replace power supply | 20:38 |
17:59 | TCS | Camilla, Jason | LVEA | YES | CO2X beam profiling | 20:59 |
18:00 | SAF | LASER HAZARD | LVEA | YES | LVEA IS LASER HAZARD | 21:00 |
18:28 | SQZ | Nutsinee | Optics Lab | Local | 21:02 | |
21:15 | SAF | Camilla | LVEA | YES | Transition to SAFE | 21:38 |
Jenne, Naoki, Louis, Camilla, Sheila
Here is comparison of the DARM CLEAN spectrum with OM2 hot vs cold. The second screenshot shows a time series of OM2 cooling off. The optical gain increased by 2%, as was seen in the past (for example 71087). Thermistor 1 shows that the thermal transient takes much longer (12 + hours) than what thermistor 2 says (2 hours).
Louis posted a comparison of the calibration between the two states, there are small differences in calibration ~1% (74913). While the DARM spectrum is worse below 25Hz, it is similar at 70 Hz where we in the past thought that the sensitivity was worse with OM2 cold. From 100-200 Hz the sensitivity seems slightly better with OM2 cold, some of the peaks are removed by Jenne's jitter subtraction (74879) but there also seems to be a lower level of noise between the peaks (which could be small enough to be a calibration issue). At high frequency the cold OM2 noise seems worse, this could be because of the squeezing. We plan to take data with some different squeezing angles tomorow and will check the squeezing angle as part of that.
So, it seems that this test gives us a different conculsion than the one we did in the spring/summer, and that now it seems that we should be able to run with OM2 cold to have better mode matching from the interferometer to the OMC. We may have not had our feedforwards well tuned in the previous test, or perhaps some other changes in the noise mean that the result is different now.
Is this additonal nosie at low frequency due to the same non-stationarity we oberved before and we believe is related to the ESD upconversion? Probably not, here's why.
First plot compares the strain spectrum from two times with cold and hot OM2. This confirms Sheila's observation.
The second and third plots are spectrograms of GDS-CALIB_STRAIN during the two periods. Both show non-stationry noise at low frequency. The third plot shows the strain spectrogram normalized to the median of the hot OM2 data: beside the non-stationariity, it looks like the background noise is higher below 30 Hz.
This is confirmed by looking at the BLRMS in the 16-60 Hz region for the two times, as shown in the fourth plot: its higher with cold OM2
Finally, the last plot shows the correlation between the ESD RMS and the strain BLRMS, normalized to the hot OM2 state. There is still a correlation, but it appear again that the cold OM2 state has an additional background noise: when the ESD RMS is att the lower end, the strain BLRMS setlles to higher values
Here is the same comparison, without squeezing. Using times from 74935 and 74834
This suggests that where cold OM2 seems better than hot OM2 above that is due to the squeezing (and the jitter subtraction Jenne added, which is also on in this plot for cold OM2 but not for hot OM2). And the additional noise with cold OM2 reaches up to about 45Hz.
After we optimized ADF demod phase in 74972, the BNS range seems better and consistently 160-165Mpc. The attached plot shows the comparison of OM2 cold/hot with/without SQZ. The OM2 cold with SQZ is measured after optimization of ADF demod phase and other measurements are same as Sheila's previous plots.
This plot supports what Sheila says in the previous alogs.
Echo-ing the above, and summarizing a look at OM2 with sqz in both Sept 2023 and Dec 2023 (running gps times dictionary is attached here).
If we compare the effect of squeezing -- there is higher kHz squeezing efficiency with hot OM2. We can look at either just the darm residuals dB[sqz/unsqz] (top), or do subtraction of non-quantum noise (bottom) which shows that hot OM2 improved the kHz squeezing level by ~0.5 dB at 1.7 kHz (the blue sqz blrms 5). This is consistent with summary pages: SQZ has not reached 4.5 dB since cooling OM2 74861. Possibly suggests better SQZ-OMC mode-matching with hot OM2.
Without squeezing, cold om2 has more optical gain and more low-freq non-quantum noise. Better IFO-OMC mode-matching with cold OM2.
In total, it's almost a wash for kHz sensitivity: heating OM2 loses a few % optical gain, but recovers 0.2-0.5 dB of shot noise squeezing.
It's worth noting the consistent range increases with SQZ tuning + improvements: even in FDS, there is a non-zero contribution of quantum noise down to almost 50 Hz. For example Naoki's adjustment of sqz angle setpoint on 12/21 74972 improved range, same for Camilla's Jan sqz tuning 75151. Looking at DARM (bottom green/purple traces), these sqz angle tunings reproducibly improved quantum noise between about 60-450 Hz.
Here are some more plots of the times that Vicky plotted above.
The first attachment is just a DARM comparison with all 4 no sqz times, OM2 cold vs hot in December vs September.
Comparing OM2 hot September vs December shows that our sensitivity at from 20-40 Hz has gotten worse since September, the MICH coherence seems lower while the jitter and SRCL coherence seem similar. The same comparison for OM2 cold shows that with OM2 cold our sensitivity has also gotten worse from 15-30 Hz.
Comparing cold vs hot, in September the MICH coherence did get worse from 60-80 Hz for cold OM2, which might explain the worse sensitivity in that region. The MICH coherence got better from 20-30 Hz where the sensitivty was better for cold OM2. The December test had better tuned MICH FF for both hot and cold OM2, so this is the better test of the impact of the curvature change.
As Gabriele pointed out with his BRUCO, 74886 there is extra coherence with DHARD Y for cold OM2 at the right frequencies to help explain the extra noise. There isn't much change in the HARD pitch coherence between these December times, but the last attachment here shows a comparison of the HARD Y coherences for hot and cold OM2 in December.
Peter asked if the difference in coherence with the HARD Yaw ASC was due to a change in the coupling or the control signal.
Here is a comparison of the control signals with OM2 hot and cold, they look very similar at the frequencies of the coherence.
Camilla, Naoki
To see the effect of ADF servo, we turned off ADF servo from 2023-12-19 22:35:25 UTC to 2023-12-20 20:25:46 UTC. The ADF itself was ON and the SQZ angle was left at the value which ADF servo ended up in the last lock. To turn off ADF servo, we changed the ADF servo flag in sqzparams to False. We also changed the nominal state of SQZ_ANG_ADJUST guardian to DOWN. We reverted these changes to turn on ADF servo.
Today Jason and I continued with the CO2 beam scan work in 74760. We did a rough alignment along the profiler, swapping the final mirror from 1" to 2" for easier alignment and checked that the profiler could see the beam. With the 7.5" FL lens on the front, the beam is too large for the profile after ~300mm. It's also quite large before the lens.
Parts stored in the same locations as last week, optics and Nanoscan still in CO2X enclosure. First optic was moved out of the beam path but we drew around the base with red pen. We should think carefully about if this data will be useful before continuing as will need ot find a different lens to successfully scan.
Rahul, Camilla, Jonathan, Erik, Dave:
At 07:33 PST during a measurement this morning the ETMX test mass was set into motion which exceeded the user-model, SWWD and HWWD RMS trigger levels. This was very similar to the 02 Dec 2023 event which eventually led to the tripping of the ETMX HWWD.
The 02 Dec event details can be found in T2300428
Following that event, it was decided to reduce the time the SUS SWWD takes to issue a local SUS DACKILL from 20 minutes to 15 minutes. It was this change which prevented the ETMX HWWD from tripping today.
The attached time plot shows the details of today's watchdog events.
The top plot (green) is h1susetmx user-model's M0 watchdog input RMS channels, and the trigger level (black) of 25000
The second plot (blue) is h1susetmx user-model's R0 watchdog input RMS channels, and the trigger level (black) of 25000
The lower plot shows the HWWD countdown minutes (black), the SUS SWWD state (red) and the SEI SWWD state (blue)
The timeline is:
07:33 ETMX is rang up, M0 watchdog exceeds its trigger level and trips, R0 watchdog almost reaches its trigger level, but does not trip.
At this point we have a driven R0 and undriven M0, which was also the case on 02 Dec which keeps ETMX rung up above the SWWD and HWWD trigger levels
The HWWD starts its 20 minute countdown
The SWWD starts its 5/15 minute countdown
+5min: SEI SWWD starts its 5 minute countdown
+10min: SEI SWWD issues DACKILL, no change to motion
+15min: SUS SWWD issues DACKILL, R0 drive is removed which resolves the motion
HWWD stops its count down with almost 5 minutes to spare.
We have opened a workpermit to reduce the sus quad models' RO trigger level to hopefully always have M0 and R0 trip together which will prevent this is the short term. Longer term solution requires a model change to alter the DACKILL logic.
During this timeline I also cleared filter history on L2_LOCK_L (very high counts before clearing) and M0_DAMP_L (no difference after clearing) details in 74889.
The channels used for the calibration measurement injections are listed in LHO:74919.
Dave, Rahul
We lost lock this afternoon and I took this opportunity to quickly implement the R0 watchdog changes. The new thresholds are given below,
ITMX R0 chain WD rms threshold - 20k counts
ITMY R0 chain WD rms threshold - 20k counts
ETMX R0 chain WD rms threshold - 18k counts
ETMY R0 chain WD rms threshold - 18k counts
I have accepted the above changes in the SDF and posted the screenshot below.
The threshold limit for ETMs is lower than that of ITMs based on the ndscope trends for the last 30days. The safety limit for ITMs seems to be around 20k and for ETMs 18k.
A more long term safety fix will be implemented in January 2024 by making some model changes.
WP 11587 Closed.
WP 11566
A thermocouple was installed on the floor between BSC2 and BSC3 to monitor temperature deviations from the LVEA floor and LVEA ambient air. Thermocouple connected to LX vacuum slow controls chassis, Terminal M21, Channel 4 (Connector G, pins 29 & 30 on side of rack).
F. Clara, G. Moreno
Using the Fluke 62 Mini Thermometer (IR) I checked the status of the input and output air on all of the Kepco power supplies at EX, EY, and CER Mezzanine per WP11586
EX and CER tested normal. Air flow is good, all supplies humming away, no odd vibrations.
At EY, ISC-YC1-TCS -24V supply looked to have a stuck fan. The supply is part of the ISC-YC1-TCS +/- 24V pair located in slot 18 of the C2 Power rack on the right hand side (RHS).
Ambient air at EY is 65F, air measured at the fan braket at 110F, and air measured output air at 80F. This supply shows -24V at 0A, low current is the reason it has not tripped already.
This pair of supplies powers pair of RF Oscillators and RF Amplifiers at EY. This is part of the Low Noise Power system thus its current draw on the +/- 24V is very low, as it is a reference.
Last checkup of supplies was September 19th, linked here ALOG72968 all supplies checked out normal. No issues were reported between then and now.
We replaced the supply with an upgraded spare with new ball bearing fan installed per WP11588 and we can hope for smooth sailing through the Christmas break.
Kepco with Failed Fan SN = S1300290
Kepco with Ball Bearing Fan SN = S1201931
M. Pirello, F. Clara
Lights and Mega CR light off, paging system unplugged, WAP off.
As TJ found last week, Robert's shaker is still connected to near HAM2, it is plugged in but powered off.
All else looked good, followed T1500386
Maintenance day ran a bit long due to an issue with the HEPI pump station at EX and an emergency power supply swap at EY, but all activities have now finished and H1 has started initial alignment.
H1 has started observing as of 22:35 UTC
Camilla, Ansel
At 17:46UTC, Camilla changed ITMX and ITMY HWS to 'sem 3', which is external trigger mode. This seemed to work, and stopped the code taking photos.
It appears that the HWS-associated combs are gone in the magnetometer witness channel after this change. Pre/post 1-hour spectra attached. (All known HWS-associated combs overlaid, just to check-- the near-7Hz is the one actually present in the "pre" spectrum which corresponds to current HWS sync frequency settings.)
Nice work Camilla and Ansel! Let's hope this solves the problem for good.
At 17:38UTC I tunred on the camera/CLink and restarted the HWS EX code which had been off since 74738.
At 17:46UTC I changed ITMX and ITMY to 'sem 3' and left them in this configuration so that they are not taking HWS data now. Now we need to write a script to read H1:GRD-ISC_LOCK_STATE_N state and adjust the camera mode to be sem 2 when we are in states < 580 (locking) and sem 3 when > 580 (locked). This is not trivial as camera computers are separate from EPICS.
Camilla, Louis We dropped out of Observing at 7am PST test the new DARM loop config: - the new DARM loop config, navigated to by going toNLN_ETMY
->NEW_DARM
, seems to be stable. We were able to sit there for several minutes without issue. - Camilla took a quick look at MICH FF settings while in 'NEW_DARM'. She will follow up with more info on that. - We lost lock again while trying to cal measurements in the NEW_DARM state. Still trying to figure out what's happening there; clearly reducing the amplitudes by 50% (LHO:74883) wasn't nearly enough. It seems we kicked the IFO hard enough for ISI_ETMX watchdogs to trip. due to the lockloss we did not get to test the DARM_RECOVER guardian state.
When the excitations started, the ETMX HEPI, ISI and SUS WDs tripped and (~5 minutes later) the hardware watchdogs for both SEI and SUS.
With all watchdogs tripped, I cleared history of the L2 LOCK L stage of ETMX as this was outputting very large numbers. M0 DAMP also seemed to be outputting very large numbers, I tried cleaning history but it seems like this was physically ust moving a LOT, plot attached.
Rahul via phone walked me though checking that ETMX is ready to be untripped: output counts changing by hundreds not 10,000's. We then reset SUS SW WD (cds > SWWD > ETMX) and untripped the SUS ETMX WD, Rahul said if it was still ringing very badly we would turn on M0 DAMP damping loops one-by-one but they already looked on so I took the GRD to DAMPED then ALIGNED.
After we got ETMX SUS back to usual, I reset the ETMX SEI SWWD and I followed the procedure to uptrip SEI (from wiki, HEPI then ISI). Once this was done Ryan took us to maintenance mode.
The new DARM configuration is shown in this plot is veery effective in reducing the ESD RMS, as shown in the plot below. The blue curve is one hour of normal configuration, showing the ESD total RMS as a function of time. The orange trace is during the test: the first few hundreds of seconds are still the nominal configuration, while between ~1000 and ~2000 seconds the new DARM configuration is running. The RMS is reduced well below the minimum of the normal configuration.
Unfortunately the was no clean data collected during this time to see the effect on the strain.
The MICHFF FM4 that we fit for cold OM2 74877 was actually very good for tis new DARM configuration green trace, much better than the FM1 we fit for the new DARM configuration 74817.
DARM didn't look great at 30-100Hz in either configuration, maybe this could be as the calibration is wrong.
The simulines measurements injected into the following channels:
H1:LSC-DARM1_EXC
H1:CAL-PCALY_SWEPT_SINE_EXC
H1:SUS-ETMX_L1_CAL_EXC
H1:SUS-ETMX_L2_CAL_EXC
H1:SUS-ETMX_L3_CAL_EXC
Injections began at approximately GPS 1387035241.
Summary of strain coherencce with all channels: https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_1386992886_STRAIN/
It looks like there is no coherence with MICH nor SRCL
At low frequency, the usual coherence with DHARD_Y
The only noticeable difference now with respect to hot OM2 is the incresed coherence with beam jitter
https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_1386992886_STRAIN_CLEAN/
Coherences for the CLEAN channel, with jitter removed
Here's a comparison of Pcal to DeltaL External and GDS Calib Strain broadband measurements with OM2 hot and cold.
File path to DTT template: /ligo/home/dana.jones/Documents/OM2_heating/OM2_hot_v_cold.xml
Dana, Louis
These broadband measurements were taken as part of the regular calibration sweeps when H1 was fully thermalized. To get the measurements, I used this command (for a sample report ID):
pydarm ls -r 20230802T000812Z | grep PCALY2DARM_BB | grep pcal
which returned:
>> pcal: /ligo/groups/cal/H1/reports/20230802T000812Z/PCALY2DARM_BB_20230727T161527Z.xml
Then to get the exact start time of the injection I used:
grep TestTime /ligo/groups/cal/H1/reports/20230802T000812Z/PCALY2DARM_BB_20230727T161527Z.xml
See this link for a list of calibration measurements. To calibrate the Delta L External and Pcal measurements I used the following files found in /ligo/home/dana.jones/Documents/OM2_heating/:
deltal_external_calib_dtt_20230621T211522Z.txt
deltal_external_calib_dtt_20230628T015112Z.txt
deltal_external_calib_dtt_20230716T034950Z.txt
deltal_external_calib_dtt_20230823T213958Z.txt
deltal_external_calib_dtt_20230830T213653Z.txt
deltal_external_calib_dtt_20231027T203619Z.txt
pcal_calib_dtt_20230621T211522Z.txt
pcal_calib_dtt_20230628T015112Z.txt
pcal_calib_dtt_20230716T034950Z.txt
pcal_calib_dtt_20230823T213958Z.txt
pcal_calib_dtt_20230830T213653Z.txt
pcal_calib_dtt_20231027T203619Z.txt
These files were generated using the following two commands (again, for a sample report ID):
pydarm export -r 20230802T000812Z --deltal_external_calib_dtt
and
pydarm export -r 20230802T000812Z --pcal_calib_dtt.
Note: For the most recent curve (23/12/10), I applied the 23/10/27 calibration TF as this was the latest valid one available.
In addition, for GDS_CALIB_STRAIN I applied a gain of 3995 in the Poles/zeros tab to convert to meters (see .xml file).
User note for calibration tab in DTT: Make sure when applying different calibration transfer functions to each curve that you set the start time appropriately—you can’t just use the same time for all of them or the system won’t know which one to choose. For each measurement, set the corresponding calibration start time to, say, 1 day before.
Here's a PCALY-to-GDS_CALIB_STRAIN broadband comparison with the OM2 in the hot state (gold trace, 2023-12-10) and in the cold state (cyan trace, 2023-12-18), see bb_hot_cold_om2.png. Both measurements were taken while the IFO was thermalized. Pcal corrections have been applied to PCALY_RX_PD_OUT. The two traces don't line up exactly but their differences are down to the percent level. Sheila is pulling up GDS_CALIB_STRAIN spectra from before and after the OM2 cooling for comparison purposes. This plot suggests that she will be able to overlay the two and compare them as long as we're not interested in making any determinations close to ~few percent level.
TITLE: 12/18 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:
- Following the EQ lockloss from earlier, H1 is back to NLN @ 0:16/OBSERVE @ 0:36 (SQZ was having trouble tuning the sqz angle due to commissioner changes, but Naoki reverted these changes so we should have no further issue)
- 0:29 - inc 4.9 EQ from Mexico
- Other than the EQ it was an uneventful night, nothing else to report
LOG:
No log for this shift.
The trouble of SQZ happened after I changed the ADF servo flag in sqzparams to False. I wanted to see SQZ drift without ADF servo, but FC IR kept unlocking after this change so I reverted it. I am not sure why FC IR got unstable without ADF servo.
We found this was due to incorrect logic in SQZ_MANAGER. If the use_sqz_angle_adjust flag is false, the node never arrives at True and goes down to SQZ_READY_IFO, unlocked FC after 120s, thinking that it's failing to reach IR_LOCKED. Edit in svn 26967.
Looks like we have an issue with the corner station seismometer having given out and caused the 'local earthquake' and lockloss(ndscope, seismon FOM, peakmon/CPSFF).
03:35UTC Back Observing