Vicky and Rahul pointed out that ZM6 looks more saturated than usual. The alignment changed last Tuesday and the LL VOLTMON (H1:SUS-ZM6_M1_VOLTMON_LL_OUTMON) has been saturated since then, plot attched. The DACs themselves (H1:FEC-43_DAC_OUTPUT_6_{8,9,1,0,11}) are not saturated hence we haven't we had verbal alarms. Unsure of the difference between VOLTMON and DAC output channels, comparism plot attached.
ZM6 is used in the SQZ ASC so we want to make sure we have range on this optic and might think about trying to offload to ZM4 or ZM5.
WP11438 HAM7 +24V IO Chassis Power Supply Swap
h1sush7 and h1seih7 are powered down, Fil is heading out to the mech room mezzanine to swap the power supply.
Fil and I went to the MER and replaced the DC power supply, the s/n are given below,
New s/n - S2001609
OLD s/n - S2001613
After replacing the power supply Dave restarted the models following which I restored the ISI and suspensions to their nominal state.
WP11438 is now closed.
Power supply replacement is complete. IO Chassis and front end computers are back online. All cards can be seen and the models are running correctly.
I untripped HAM7 SWWD and handed it over to the control room for recovery.
Broken Kepco power supply (removed) | S2001613 |
New Kepco power supply (installed) | S2001609 |
TITLE: 09/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 12mph Gusts, 11mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.59 μm/s
QUICK SUMMARY:
Workstations were updated and rebooted. This was an OS update. Conda packages were not updated.
TITLE: 09/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 150Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY:
High microseism made it very hard to relock the detector once we lost lock at 09/26 01:26UTC(73096). After losing lock on our way up 8 different times, I put the detector in DOWN for half an hour in the hopes that the microseisms would start trending down and allow us to lock, and that next lock attempt we made it up to NOMINAL_LOW_NOISE.
Once we got back to NLN, DIAG_MAIN was saying that the ISS pump was off, and we took OPO to LOCKED_CLF_DUAL_NO_ISS and then requested SQZ_MANAGER to FREQ_DEP_SQZ, Vicky and I noticed that the OPO still couldn't catch, so we lowered the opo_grTrans_setpoint_uW from 80 to 70 for the night and reloaded OPO (based on 70050). OPO caught and FDS turned on and we automatically went into Observing, but since the temp setpoint hadn't been adjusted (like it says to do in the log), our range was low so we briefly went back out of Observing to try and adjust it. We got it the best that we could, accepted SDF diffs, and got back into Observing at 06:59UTC.
Microseism does look like it is finally starting to turn around(attachment2).
LOG:
23:00UTC Detector Observing and Locked for 35hours, useism very high at 0.67um/s due to storm off the coast of Washington
01:26 Lockloss (73096)
01:57 Lockloss from ACQUIRE_DRMI_1F, starting INITIAL_ALIGNMENT
02:48 INITIAL_ALIGNMENT complete, heading to NOMINAL_LOW_NOISE; useism = 0.83um/s
02:53 Lockloss from ALIGN_RECYCLING_MIRRORS
02:56 Lockloss from FIND_IR
03:04 Lockloss from ACQUIRE_DRMI_1F
03:12 Lockloss from ENGAGE_DRMI_ASC
03:17 Lockloss from FIND_IR
03:23 IR(DIFF) was not found - it found it but passed through it
03:30 Lockloss from CHECK_IR while I was adjusting ALS DIFF
03:33 IR was not found - I adjusted ALS DIFF and got it
04:13 Lockloss from TRANSITION_FROM_ETMX
04:28 Lockloss from DHARD_WFS, putting detector in DOWN for 30mins
05:04 Attempting relocking
05:50 NOMINAL_LOW_NOISE
- SQZ_ISS Pump is off
06:32 Observing
06:43 Out of Observing to adjust OPO temp setpoint for the new opo_grTrans_setpoint_uW
06:59 Back to Observing
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
0:00 | PCAL | Tony | PCAL Lab | y(local) | Getting stuff ready for Tuesday | 00:30 |
06:43 | SQZ | Oli | CR | n | Adjust SQZ OPO temp | 07:02 |
Lockloss @ 09/26 01:26UTC after a 37 hour lock :(
Back Observing 09/26 06:59UTC
Microseism is currently very high (moving between 0.73 and 0.93 um/s), and we have had threefive (we had two more locklosses while I was writing this) locklosses at different states (ALIGN_RECYCLING_MIRRORS, FIND_IR, ACQUIRE_DRMI_1F, ENGAGE_DRMI_ASC, FIND_IR) after running an initial alignment, so it's possible we won't be able to lock the detector for a bit due to the high microseism. I'll let it try a couple more times but we might have to wait for the ground to calm down. Luckily wind is low (10mph) and looks to be decreasing.
We almost made it up to NLN but lost lock at TRANSITION_FROM_ETMX, and lost lock once before that at CHECK_IR and just now again at DHARD_WFS so I put the detector in DOWN and set Observatory Mode to MICROSEISM and will try locking again in ~30 minutes with hopefully better luck and lower microseism.
TITLE: 09/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
PSL_ISS reflected power is still low
NonSens Noise measurement while LLO is relocking.
17:00 UTC H1:OAF-NOISE_WHITENING_GAIN was brought to 1
17:42 UTC H1:OAF-NOISE_WHITENING_GAIN was brought back to 0
and we went back to Observing
19:00 UTC EV Charging Station repairman arrived to Install a new modem for the Charging station.
MicroSeism Finally platteau'd about 2 hours ago.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
15:00 | FAC | Karen | Optics-Lab | N | Technical Cleaning | 15:27 |
15:35 | FAC | Tyler | AirHandlerU1 | N | Air temp check | 15:41 |
15:52 | FAC | Kim | Mid X | N | Technical cleaning | 16:49 |
15:52 | FAC | Karen | MidY | N | technical Cleaning | 17:12 |
16:58 | Commish | Jenne | Remote | N | NonSens gain increased for 1 hour | 17:58 |
17:14 | FAC | Chris | CER | N | getting a \ | 17:24 |
17:53 | FAC | Cindi | Woodshop | N | Technical Laundry | 18:30 |
18:00 | FAC | Karen | MID Y | N | Putting an Extension cord back. | 18:08 |
18:30 | FAC | Cindi | OSB Receiving | N | unloading cardboard | 18:44 |
TITLE: 09/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 150Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 6mph Gusts, 4mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.67 μm/s
QUICK SUMMARY:
Locked for 35hours! useism is HIGH because of the storm off the Washington/Canada coast (73084) but it looks like it's finally plateau'd.
FAMIS 19995
Only major event of note is the laser going down from a power glitch three days ago; everything seems to have recovered as expected.
EDIT: Forgot to attach the trends.
Mon Sep 25 10:10:34 2023 INFO: Fill completed in 10min 30secs
Jordan confirmed a good fill curbside.
The review team for the NonSENS online cleaning asked both observatories to gather 1 hour of otherwise undisturbed time with NonSENS subtraction engaged, so that they can do some tests before approving the cleaning for online use.
Since LLO is unlocked, Tony just turned on the NonSENS subtraction. This has us in Commissionining. We'll stay this way for 1 hour, or we'll go back to Observe when LLO is back, whichever comes first.
Richard asked me this morning to look at air flows at EX and to possibly turn the flow down. I have adjusted the flow from 8300 to 7000 CFM. This should not have any effect on the temperature.
TITLE: 09/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 8mph Gusts, 6mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.44 μm/s
QUICK SUMMARY:
Micro Seism is rising rapidly due to this storm off the coast.
h1susauxh2 DAQ CRC errors were cleared at 10:00 PDT while H1 was out-of-observe.
TITLE: 09/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: We're Observing and have been Locked for 19hours. Quiet shift with the only thing happening being elevated dust in the PSL for a bit and Ryan S turning the PSL watchdogs back on. PSL ISS Diff power is still low, but can wait until tomorrow or the next lockloss.
23:00UTC Detector Observing and Locked for 11hours
01:00 Ryan S turned PSL Watchdogs on - we stayed Observing (73081)
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
01:00 | PSL | Ryan S | remote | n | Turning PSL Watchdogs on | 01:01 |
Right after lockloss DIAG_MAIN was showing:
- PSL_ISS: Diffracted power is low
- OPLEV_SUMS: ETMX sums low
05:41UTC LOCKING_ARMS_GREEN, the detector couldn't see ALSY at all and I noticed the ETM/ITMs L2 saturations(attachment1-L2 ITMX MASTER_OUT as example) on the saturation monitor(attachment2) and noticed the ETMY oplev moving around wildly(attachment3)
05:54 I took the detector to DOWN, and immediately could see ALSY on the cameras and ndscope; L2 saturations were all still there
05:56 and 06:05 I went to LOCKING_ARMS_GREEN again and GREEN_ARMS_MANUAL, respectively, but wasn't able to lock anything - I was able to go to LOCKING_ARMS_GREEN and GREEN_ARMS_MANUAL without the saturations getting too bad but both ALSX and Y evenually locked for a few seconds each and then unlocked and gone to basically 0 on the cameras and ndscope(attachment4). ALSX and Y sometime after this went to FAULT and were giving the messages "PDH" and "ReflPD A"
06:07 Tried going to INITIAL_ALIGNMENT but Verbal kept calling out all of the optics as saturating, ALIGN_IFO wouldn't move past SET_SUS_FOR_ALS_FPMI, and the oplev overview screen showed ETMY, ITMX, and ETMX moving all over the place (I was worried something would go really wrong if I left it in that configuration)
I tried waiting a bit and then tried INITIAL_ALIGNMENT and LOCKING_ARMS_GREEN again but had the same results.
Naoki came on and thought the saturations might be due to the camera servos, so he cleared the camera servo history and all of the saturations disappeared (attachment5) and an INITIAL_ALIGNMENT is now running fine.
Not sure what caused the saturations in the camera servos, but it'll be worth looking into whether the saturations are what ended up causing this lockloss.
As shown in the attached figure, the output of camera servo for PIT suddenly became 1e20 and the lockloss happened after 0.3s. So this lockloss is likely due to the camera servo. I checked the input signal of camera servo. The bottom left is the input of PIT1 and should be same as the BS camera output (bottom right). The BS camera output itself looks OK, but the input of PIT1 became red dashed and the output became 1e20. The situation is same for PIT2, 3. I am not sure why this happened.
Opened FRS29216 . It looks like a transient NaN on ETMY_Y camera centroid channel caused a latching of 1e20 on the ASC CAM_PIT filter modules outputs.
Camilla, Oli
We trended the ASC-CAM_PIT{1,2,3}_INMON/OUTPUT channels alongside ASC-AS_A_DC_NSUM_OUT_DQ(attachment1) and found that according to ASC-AS_A_DC_NSUM_OUT_DQ, the lockloss started(cursor 1) ~0.37s before PIT{1,2,3}_INMON read NaN(cursor 2), so the lockloss was not caused by the camera servos.
It is possible that the NaN values are linked to the light dropping off of the PIT2 and 3 cameras right after the lockloss, as when the cameras come back online ~0.2s later, both the PIT2 and PIT3 cameras are at 0, and looking back over several locklosses it looks like these two cameras tend to drop to 0 between 0.35 and 0.55s after the lockloss starts. However the PIT1 camera is still registering light for another 0.8s after coming back online(typical time for this camera).
Patrick updated the camera server to solve the issue in alog73228.
Based on TJ's log 72857 about recent FC alignment drifts, I wanted to check out the situation with possible scattered light from the filter cavity since alignments can drift around (worth investigating more), and I wanted to check the CLF fiber polarization (this was basically fine).
Summary screenshot shows some measurements from today. I'm wondering if we might be closer than I realized to filter cavity backscatter. With an excitation that is ~5-10x greater than the ambient lsc control to FC2, it created scatter about 2x above darm at 100 Hz. Based on previous (higher-freq) measurements (LHO:68022) and estimates (LHO:67586), I had thought ambient LSC control was about 10-fold below DARM; this suggests we are within a factor of ~5? Though, there is a lot of uncertainty (on my end) of how hard we are driving FC2 given the suspension/loop roll-offs/etc, so I need to think more about the scaling between measured scatter and the excitation.
Strange line in FC-LSC error signal -- it seems to wander; I've seen it (or things like it) sometimes in other SQZ-related signals; but I haven't figured out where it comes from yet. It can be easily seen on SQZ summary pages (sqz > subsystems tab) since Derek and Iara helped us it up (thank you Detchar!!). I don't see it in CLF_ISS but sometimes in other error signals. Not clear to me that this is an issue if the peak is >100 Hz, but if it drifts to lower frequencies and this line is real/physical (not some random artifact), it could be problematic. The peak amplitude seems large enough that if it were in-band of the FC2 suspension and controls, it could plausibly get injected as real FC length noise and drive some measurable backscatter.
For the excitation -- I used the template from LHO:68022 but ran it to lower frequencies, in-band of FC-LSC. Compared to the FC error signal (specifcally H1:SQZ-FC_WFS_A_I_SUM_OUTPUT which is the input sensor to FC_LSC_DOF2_IN1), this DTT injection of 30,000 counts increased the integrated RMS of the in-loop error signal at 10 Hz by about 9-fold (= 3375 (w/excitation) / 387 (ambient), measured from dtt rms cursors). I injected into the fc-lsc loop at H1:SQZ-FC_LSC_DOF2_EXC, with various amplitudes (like 30k), and filter = butter("BandPass",4,10,300); this should then go to FC2_L for suspension feedback. I'm not sure that I'm using the best witness sensors for actual length noise driven in this excitation, but wasn't able to totally figure it out in time.
With this excitation going, I tried to walk the alignment to see if there was an alignment that minimizes backscatter, but I didn't figure this out in time. I tried to walk ZM2 with beam spot control off, and then set the QPD offsets where it landed. This was probably the wrong approach, since I wasn't able to then set the QPD offsets in time; maybe I should have walked the FC-QPD offsets with full ASC running at higher gain, since this loop is so slow. Might be worth trying this again for a bit; with the injection running, I wasn't sure if I was able to minimize scatter by walking ZM2 (p/y/ maybe psams?), but there were a couple directions in both pitch and yaw that looked promising.
In the end, SDF diffs for ZM2 were accepted (ZM2 is not under asc-control), and I accepted the beam spot position change in the FC QPD Pitch offset (from 0.07 --> 0.08) while reverting the yaw change that I didn't figure out in time. I don't anticipate much overall change in the squeezer after these tests.