J. Kissel, G. Vajente ECR E2400116 IIET 30863 WP 11797 After yesterday's infrastructure changes to the H1 SUS OMC front-end model (LHO:76856 and LHO:76894), which moved the OMC ASC control signals - from going . straight from the OMC ASC 2k model . through the SUS-OMC_M1_ASC filter bank for plant compensation, and . then out to EUL2OSEM matrix for basis transformation to the DAC - to going . from the OMC ASC 2k model . through the SUS-OMC_M2_ISCINF filter bank (for proper 2k to 16 kHz anti-imaging), . through the SUS-OMC_M1_LOCK filter bank (for the same filtering that was done by the former M1_ASC bank), . through the the SUS-OMC_M1_DRIVEALIGN matrix (though configured at the moment to no do any off-diagonal decoupling), and . *then* out to EUL2OSEM matrix for basis transformation to the DAC there was some worried that the excess noise between 10 and 100 Hz might be this infrastructure change. It's not, as far as we can tell, responsible for and change in the noise in DARM. In LHO:76928, Gabriele posted an ASD of the SUS-OMC_M1_MASTER_OUT_DQ channels to confirm that the drive to the OMC SUS's DAC is the same (the thought being that if the DAC request is drastically different, then we should be suspicious of the infrastructure change). Here, in the attached ASD I show a different perspective -- the input to the former M1_ASC bank compared with the current M1_LOCK bank in pitch and yaw, and then showing those channels' coherence with DARM. I compare 25 averages (0.02 Hz frequency resolution, 50% overlap) two times, RED 2024-04-02 6:00 UTC Pre-change (in the middle of a good overnight lock stretch; 2024-04-01 23:00 PDT) BLUE 2024-04-03 18:05 UTC Post-change (in the middle of a good lock stretch this afternoon, though in the middle of SQZ tuning of ZM alignment) This ASD also focuses on where the control signal is real signal (from the very slow loop design -- see LHO:65861). While the character of the ASD control signal and coherence with DARM has changed around the (presumably OMC SUS) resonances, this perspective corroborates that we don't see substantial change. *Maybe* the change in character around the resonances changed because I misinterpreted the former infrastructure, and somehow we *were* using the confused DRIVEALIGN gain values from LHO:47488, but I think not.
Notes from locking this morning (with Alignment woes and Transition from ETMx locklooses)
After spending 3hrs trying to get through an alignment yesterday afternoon, and since H1 was having issues after this morning's lockloss and multiple Initial Alignment cycles (and also after talking with TJ [owl shifter]), decided to REVERT all suspensions (except squeezer) to a time before last night's lock.
Chose time when ISC LOCK was in the Alignment state (#11) and picked a time within a minute of the end of when it was in this state. (Still not sure if that is the best way to pick a time, because how would one know if ALL sus were in an "aligned" state.) So---I reverted to the following time:
Alignment #1: Going for a Manual Alignment to slowly go through each alignment state since some states are broke.
Lock#1: Selected PRMI (locked within 15sec), but had a lockloss at Transition From ETMx.
Then there were several early-state locklosses.
Had another Transition From ETMx lockloss at 1719utc (note: during this lock and before the lockloss, Jim brought the BRSx back in-loop (at 1656utc)---we hadn't been using it for the last day or so.).
Finally, H1 made it to NLN at 1805utc.
After lockloss from a 2hr44min lock, H1 had issues with early-state locklosses. It also ended up looking misaligned via a dead DRMI acquisition. So ran CHECK MICH FRINGES + PRMI (which needed PRM tweak). DRMI also needed SRM yaw-tweaks. After this H1 returned to our old friend Transition From ETMx and had a lockloss here (3rd time this has happened today).
For recent locking, the Y-arm was not locking. So it automatically went through a round of INCREASE FLASHES (which took a long time) and resulted in a not very great alignment with flashes much lower than what one could get by tweaking by hand when Unlocked---which is what I ended up doing. (It feels like with winds above 10mph the green arms have been having issues sadly.)
We got some no squeeze quiet time almost an hour into lock this morning and our low frequency noise looks worse than yesterday - so impacting our range.
Sensor correction is on (confirmed by Corey).
Not sure what has changed.
Template is saved as /ligo/home/jennifer.wright/Documents/Noise_DARM/20240403_DARM_comp_sqz_no_sqz2.xml
I compared this with NO SQZ quiet times from yesterday, 17th March, and December 2023 during O4a. These other times were all after thermlisation.
https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_1396204818_GDS_CALIB_CLEAN/
Low frequency coherence with CHARD_P and HAM1
PRCL is high, but we've seen something at this level even before
No smoking gun for 30-40 Hz
The spectrogram attached here (whitened to the median of the "good" DARM spectrum from yesterday) shows hints that the excess noise is not very stationary. BruCo and coherence might not give us much info, but I'm running it nevertheless
OMC suspension MASTER_OUT signals look unchanged after Jeff's modification
More corroborative evidence that changes to the OMC ASC control signal path through the OMC SUS is not causing the excess noise in DARM -- LHO:76929.
It does seem possble that we are clipping on the LSC pop diode. The attached screenshot shows that compared to the lock April 2nd we have very smiilar arm circulating powers, but 0.3% less power on LSC POP.
We've previously seen clipping on POP cause low frequency nosie like this: 74641 69931 The second screenshot shows what these trends looked like at that time, it was much more clear that we had a clipping problem then.
Wed Apr 03 10:07:57 2024 INFO: Fill completed in 7min 53secs
Jordan confirmed a good fill curbside.
cds-crtools was updated to version 4.11 in the CDS conda environment.
The major changes are:
FOTON_WARN_RHP_ZEROS=true.
The criterion for detecting a zero in the right-hand plane has been relaxed. Zeros
that are ostensibly on the imaginary axis, but due to precision issues are slightly in the right-hand plane will no longer be flagged.A complete change history for all crtools can be found here:
WP11743 SUS DACKILL removal and WD consolidation
Jeff, Oli, Dave
New models were install for h1susim, h1sushtts, h1sussqzin, h1sussqzout, h1susifoout. No SEI model changes were needed, IOP models were not restarted so no SWWD bypass was necessary. DAQ restart was required.
Beckhoff OPLEV upgrade
Fernando, Daniel, Dave:
New OPLEV code was installed on the Beckhoff slow controls, resulting in additional 20 slow channels to H1EPICS_ECATAUX[CS,EX,EY]. EDC+DAQ restart was required.
PSL Neolase Beckhoff code change
Patrick, Ryan S, Dave:
The PSL Diode room Beckhoff code was modified, resulting in one additional channel (H1:PSL-LASER_PDWD) in H1EPICS_PSLNEOLASE.ini. EDC+DAQ restart was required.
DAQ Restarts
Jeff, Dave:
The DAQ was restarted twice. The first around 09:30 immediately after the SUS model changes, and then a second time with an EDC restart around 11:30 to support the Beckhoff changes.
GDS Broadcast List Change:
The h1susifoout change resulted in the removal of 6 fast channels (256Hz) from the frame, of which 2 were being sent DMT via the GDS broadcaster. ECR2400116 was created and approved to cover this removal.
--: fast channel H1:SUS-OMC_M1_ASC_L_IN1_DQ removed from the DAQ
-!: fast channel H1:SUS-OMC_M1_ASC_P_IN1_DQ removed from the DAQ ***GDS-CHAN***
--: fast channel H1:SUS-OMC_M1_ASC_R_IN1_DQ removed from the DAQ
--: fast channel H1:SUS-OMC_M1_ASC_T_IN1_DQ removed from the DAQ
--: fast channel H1:SUS-OMC_M1_ASC_V_IN1_DQ removed from the DAQ
-!: fast channel H1:SUS-OMC_M1_ASC_Y_IN1_DQ removed from the DAQ ***GDS-CHAN***
To expediate the broadcast list change, I hand editied the gds_broadcast.ini file on h1daqgds[0,1]. This needs to be made permanent in puppet.
Three fast channels were added to the frame:
++: fast channel H1:SUS-OMC_M1_LOCK_L_IN1_DQ added to the DAQ
++: fast channel H1:SUS-OMC_M1_LOCK_P_IN1_DQ added to the DAQ
++: fast channel H1:SUS-OMC_M1_LOCK_Y_IN1_DQ added to the DAQ
Tue02Apr2024
LOC TIME HOSTNAME MODEL/REBOOT
09:20:16 h1sush2b h1susim <<< SUS MODEL RESTARTS
09:20:36 h1sush2b h1sushtts
09:21:39 h1sush56 h1susifoout
09:22:00 h1sush56 h1sussqzout
09:22:49 h1sush7 h1sussqzin
09:31:24 h1daqdc0 [DAQ] <<< FIRST DAQ RESTART
09:31:36 h1daqfw0 [DAQ]
09:31:36 h1daqtw0 [DAQ]
09:31:37 h1daqnds0 [DAQ]
09:31:45 h1daqgds0 [DAQ] (NEW BRD LIST)
09:34:35 h1daqdc1 [DAQ]
09:34:48 h1daqfw1 [DAQ]
09:34:48 h1daqtw1 [DAQ]
09:34:51 h1daqnds1 [DAQ]
09:34:59 h1daqgds1 [DAQ] (NEW BRD LIST)
09:35:58 h1daqfw0 [DAQ] <<< FW0 Spontaneous restart
11:15:36 h1sush7 h1sussqzin <<< 2ND SQZIN RESTART, PARTS FOUND TO BE MISSING
11:30:51 h1daqdc0 [DAQ] <<< 2ND DAQ RESTART FOR BECKHOFF AND SUSSQZIN
11:31:01 h1daqfw0 [DAQ]
11:31:01 h1daqtw0 [DAQ]
11:31:02 h1daqnds0 [DAQ]
11:31:09 h1daqgds0 [DAQ]
11:31:41 h1susauxb123 h1edc[DAQ] <<< EDC RESTART
11:32:19 h1daqgds0 [DAQ] <<< 2ND GDS0 RESTART NEEDED
11:35:01 h1daqdc1 [DAQ]
11:35:14 h1daqfw1 [DAQ]
11:35:15 h1daqtw1 [DAQ]
11:35:18 h1daqnds1 [DAQ]
11:35:26 h1daqgds1 [DAQ]
11:36:01 h1daqgds1 [DAQ] <<< 2ND GDS1 RESTART NEEDED
11:51:00 h1daqnds1 [DAQ]
Yesterday 4-2-2024 the HVAC pre-filters at each of the 4 out station buildings along the arms were replaced with new pre-filters. The new pre filters are the same size (24"x24"x2") and filtration level (MERV-13) as the old pre-filters that were taken out. During inspection of the pre-filters an inspection was done on the "Main Filter Cartridge" which is the harder, more permanent, deeper, and heavier duty material type cartridge. It's installed in front of the pre-filter that is used for air filtration in the HVAC system. These are nearing the end of their life cycle. These will likely be replaced in the coming couple months.
Alerted around 645 PT about initial alignment not completing. I found it with MICH not locking, but flashes seemed a bit off. I tried realigning via the camera and trying to maximize the flashes in AS_A. Even after getting a good spot on camera and having good flashes, it would diverge when tring to lock. MICH dark was the same. I tried to skip this state but SR2 alignment wouldn't even lock, and trying it by hand seemed that there was something very off.
Richard had told me that he set a node to INIT around 630PT, trending back it looked like in Input_Align. We've been having issues with this state recently, I guess that makes sense. I then tried this state and it locked without issue. I then tried PRC, no issues. Back to MICH, same issues as before. Tried skipping it with a well aligned spot on camera, and ran into the same issues with SR2 alignment.
I'm handing off to Corey with the suggestion to restore alignment to the last alignment Oli did and then trying an alignment since I was able to lock with this earlier.
TITLE: 04/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 11mph Gusts, 9mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.23 μm/s
QUICK SUMMARY:
H1 lost lock about 3.5hrs ago (1238utc), and TJ (owl shift-er) has been trying to help H1 with an alignment, BUT the Initial Alignment woes continue. Strangely, Input Align appeared to work for him, but every other (atleast MichBright and SRC) were not completing for him.
Will try an overall alignment revert to the alignment Oli/TJ had last night. And from that will run an alignment to see if Initial Alignment steps still have issues. (if they do, I'll skip the alignment and try locking with the reverted alignment.
Enviornmentally all is quiet after the windy and shaky night.
Oli handed off the IFO with the wind keeping ALS from locking, but they made it through an initial alignment earlier. Once the wind calmed down a bit ~30min later, it was able to keep ALS locked and then went right up without intervention.
TITLE: 04/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Relocking from when I showed up went quickly, but after the earthquake knocked us out it's been impossible to lock due to the ground motion and the rising high winds :(
LOG:
23:30 Reached NOMINAL_LOW_NOISE
23:41 Observing
00:11 Lockloss due to large earthquake from Taiwan (76897), holding in IDLE
Relocking
04:45 Untripped all watchdogs, attempting to lock green arms before starting an inital alignment
- Green arms are super wiggly still from the earthquake/wind/both but I finally got them both
05:18 Started MANUAL_INITIAL_ALIGNMENT
05:24 Skipped INPUT_ALIGN_OFFLOADED due to same issues everyone else has had recently (76870, 76843)
- I couldn't even get it to ACQUIRE_XARM_IR
05:41 Initial alignment complete, relocking
- Having trouble locking ALSY due to high wind at EY - you can see the spot swinging on camera :(. Flashes look great though!
05:56 Going to sit in DOWN for 10 mins and rest the quads - wind just broke past 40mph
06:02 Tried to take advantage of a slight drop in wind speeds, but didn't work
06:16 Went back to sitting in DOWN
06:33 Trying locking again
06:35 Lockloss during LOCKING_ALS
06:48 Back to DOWN after multiple locklosses during LOCKING_ARMS_GREEN and LOCKING_ALS
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
22:22 | PCAL | francisco, rick | pcalLab | y | Pcal work - rick out 0215 | 02:27 |
Ran the data for the In Lock SUS Charge Measurements that ran this morning. Like Camilla said (76895), the excitations only worked for the ETMs, so although I am attaching all four plots, note that the last plot point for the ITMs is March 13th (ETMX, ETMY, ITMX, ITMY).
Adding these measurement results also calculated with new scripts. As far as I can see, similar trends show up with the caveat that signs of variables are sometimes different. By the way this makes me wonder a bit. We'd assumed that ETMX got charged up during the break, but the absolute value of beta and beta2 (which determine linear force component strength - see eq.3 in T1700446) for ETMX in last two measurements go down in both old and new calculation. Isn't it the opposite of what we'd expect?
We've been sitting in IDLE since the earthquake rolled through three hours ago. Ground motion is still too high to lock.
Functionality tests were performed on the LVEA turbo stations. The details: X-manifold: Bearings: 100% Turbo hours: 1950 Scroll hours: 1946 The recently installed separated water lines were also tested and found water-tight. Y-manifold: Bearings: 100% Turbo hours: 951 Scroll hours: 2279 OMC: Bearings: 100% Turbo hours: 5964 Scroll hours: 5903
Lockloss at 04/03 00:11UTC from Taiwanese earthquake that Jamie experienced in person