TITLE: 04/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C - OWL CANCELLED
SHIFT SUMMARY:

IFO is LOCKING and in ENGAGE_DRMI_ASC

• Had an hour and 16 minutes of terrible range (125 MPc) observing time following fruitless bad alignment post-maintenance troubleshooting. Details of this troubleshooting are listed in alog 77368 and its comments.
• Naoki and Sheila tried improving the squeezing from around 4:25 - 5:25 UTC to no/poor avail.
• More work will be done tomorrow to see why H1 is in such a bad alignment. 
• Jenne okayed keeping the Remote OWL off due to these issues unless "significant progress" was made - I was going to turn it on if we had stayed in observing but given the short observing time, bad range, short lock time and unstable auto-locking post lockloss, I will not be enabling the remote OWL. This is not mentioning the 21 (recorded) locklosses it took to reach this NLN Observing State
• IY Violin 6 (The very annoying one that needs 5 to damp with it) has been extremely high throughout this whole 16 hr+ lock acquisition
• Find SDF Diff Screenshots Below - there were 8 screens due to new offset settings that Jenne okayed for SDF change acceptance.

5:24 UTC - Reached NLN but waiting as SQZ team (Naoki) tries to better align squeezer

5:40 UTC - Observing

6:39 UTC - Lockloss DRMI

LOG:

IFO is in LOCKING_ARMS_GREEN and still locking after maintenance.

There is a LOT of ongoing and past troubleshooting mostly summarized by Jenne in alog 77368. Since this was posted, I looked at the HAM6 Picomotors and determined that there was no suspicious activity (esp in the on/off switching) since locking issues began (approx 15 hrs ago now).

We just got to OMC Whitening but the violins were horrendous. We stayed there for 30 minutes before an unknown lockloss DRMI. Sheila is on TS helping with SQZ alignment/instructions with SQZ alignment.

[Jenne, Ibrahim, RyanS, Sheila, Robert, TJ, Elenna, Jennie, Oli, Jim, others]

Something is different with the IFO today, and it's not great.  Since we haven't pinpointed what the problem is, we don't really know if it is related to our mediocre range last night and struggles to relock after that, or if it is new from maintenance day.  So far, the solution to get back to (almost) NLN has been to put in *massive* offsets in AS_C.  This did enable us to get locked (and avoided the ASC ringup that was causing locklosses this afternoon), but it left us with a large scatter shelf in DARM.  Robert had the good suggestion that we see if, once locked, we could walk the offsets back to their nominal values of zero; doing this caused an ASC ringup, and it's probably the same thing that we'd been seeing throughout the day.  So, indeed going toward the nominal offset of zero for pit and yaw on AS_C is not an okay place right now.

We began to narrow in on AS_C since, during initial alignment, it is in-loop for SR2 and SRY alignments, and it was causing SR2 to be pulled very far in yaw.  We were seeing this visually on the AS_AIR camera, which looked really unusually terrible. In the end, Sheila hand-aligned SRY, and then put offsets into AS_C such that the ASC now servos to that point.  But, the offsets used are +0.46 in pit and -0.85 in yaw, so really, really big. However, with these in place, we were able to let DRMI ASC run, and it ran fine. 

Since that worked (the large AS_C offsets), we let the IFO relock the rest of the way, and it kept on working. Per a suggestion from Elenna from earlier in the afternoon, after we completed REDUCE_RF45 I manual-ed to ADJUST_POWER and increased the power 5W at a time (waiting for ASC to mostly converge between steps) until we were at 60W, then I manual-ed back to LOWNOISE_ASC.  After that, I just selected NomLowNoise and guardian finished things.  We ended up 'stuck' in OMC_WHITENING, although the violins were coming down with the damping that guardian was doing.  It was around here that I tried ramping down the AS_C yaw offset with 10 sec ramp times to see if it would reduce the scatter shelf that we saw in DARM.  See first attachment, with the scatter shelf circled.  My first step was to -0.80, second step was to -0.70, and we had an ASC ringup and lockloss at this step.  I wasn't sure after the first step, but definitely as we were ramping to the second step (before we lost lock) RyanS, Robert, and I all agreed that the scatter shelf was getting worse, not better.  But, we lost lock before I could put the offset back.

We still don't understand *why* we need these large offsets in AS_C, just that we do.  Since I hadn't yet SDF-ed them, we had 2 locklosses from ENGAGE_DRMI_ASC when the nominal zero-offsets were used in AS_C.  I have since saved the AS_C offset values, and the offset switch being on, in both the safe.snap and observe.snap SDF files.  The second and third attachments show these.

We've been trying to think through what from maintenance day could possible have had anything to do with this, and the only things we've come up with are the h1asc reboot and the grounding of SUS racks.  From Jeff's alog about the h1asc changes, it seems quite clear that those are just additions of monitor points, and that shouldn't have had any effect. While we were unlocked after our one successful attempt, Robert went into the LVEA and un-grounded those two racks that Fil grounded earlier today.  Robert is quite sure that he's checked the grounding of them in the past without any consequence to our ability to lock, but just in case we've got them undone for tonight. So far, that does not seem to have changed the fact that we need these crazy offsets in AS_C.  Just in case, Robert suggests we consider rebooting h1asc tomorrow to back out other changes that happened today (even though those shouldn't have anything to do with anything).

For right now (as of ~7:20pm), my best plan is to see if we can get to Observing, and should a GW candidate come through, have the detchar and parameter estimation folks exclude all data below ~80 Hz. This is a really very poor solution though, since this huge scatter shelf will be quite detrimental to our data quality and could cause our search pipelines to trigger on this scattering. 

Related thought, but not yet thoroughly investigated (at least by me), is whether our problems actually started sometime yesterday or last night, and aren't at all related to maintenance. As partial 'evidence' in this direction, I'll note that kappa_c has been dramatically low for the last two Observing segments last night.  We haven't gotten all the way to NLN tonight yet (OMC_WHITENING didn't finish before we lost lock), but kappa_c looks like it might be even lower now than yesterday.  The 4th attachment shows kappa_c for last night's locks (screenshot taken before we locked today).  So, maybe we've had some clipping or severe degredation of alignment in the last day or so.   Jim checked all of the ISIs, and nothing seems suspicious there.

Last thing - since we had run the dark offsets script earlier this afternoon (probably was a red herring) and I saved all of the SDFs in the safe.snap, we will have diffs for Observe.snap.  All diffs that have _OFFSET in them should be accepted (and screenshot-ed). 

Because of the locking problems, I disconnected the grounds that were installed earlier today (77350).

Corey, Robert

Derek reported that S240420aw was likely caused by scattering noise so I looked into the problem. The figure shows that a large seismic pulse (~2 orders of magnitude above background) started something swinging at 1.44 Hz, with a Q of about 300. The scattering shelf's cutoff slowly dropped in frequency over ten minutes,  so there were plenty of chirps reaching different frequencies.  It is tempting to guess that the source of scattering is a TMS, which have transverse resonances of about 1.4 Hz, but I checked and the Qs of TMS motion were much lower than the Q of 300 of the scatter source.  The time of the seismic pulse matches that of the M2.8 earthquake in Richland on Friday night, 24 surface km from the site and 8 km deep. There is quite a bit of light scattered, but at normal ground motion levels I would expect the scattering noise shelf to reach only about 5 Hz, so I don’t think this scattering noise source affects us during normal operation. Andy L. and Beverly B. noticed similar scattering noise back in 2017 ( 37947 ) for a smaller nearby quake.

Jennie W, Elenna, Jenne, TJ

We have been having trouble locking and keep falling out during the MAX_POWER state of ISC_LOCK as we power up (a and b are examples showing MICH and DHARD during two of these four locklosses).

During some of these we noticed that MICH or DHARD could be ringing up.

We measured MICH open loop gain with Elenna's template which I have saved a version of in /ligo/home/jennifer.wright/Documents/ASC/MICH_P_OLG_broadband_shaped.xml. We foudn the gain to be about 10% too low so increased the loop gain from -2.4 to -2.7 in the loop servo. We also measured DHARD open loop gain with the template in this folder DHARD_2W_P_OLG_broadband_shaped.xml and found it looks normal.

Investigations ongoing...

TITLE: 04/23 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 8mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

Still in post-maintenance LOCKING

Attempting to re-lock after maintenance though facing some issues as per alog 77363.

TITLE: 04/23 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: Started an initial alignment just after 1130 local time. Had no issues during the alignment, but when we tried to start main locking we would lose lock around the power up. This happened 4 times in total. Each lock loss would ring the violins up so we would have to damp them at 2 W. After the 3rd one during the time we were waiting for violins to damp, we ran some OLG of MICH P and DHARD P (the two channels we saw extra motion before the lock loss). This has us bump up the MICH P gain, but we lost lock again. We then decided to run the dark offsets and another initial alignment. Were currently noticing that initial alignment looks way off in MICH Bright and SR2 alignment. Working on that.

LOG:

• 1952 Lock loss @Max Power (55W)
• 2026 Lock loss @Max Power (40W)
• 2129 Lock loss @Max Power (30W)
• 2230 Lock loss @Max Power (30W)                      

TJ, Jenne, Jennie

We ran the dark offset script today, to see if there were hidden dark offsets that got reverted during today's h1asc boot.  TJ pointed out that we'd been having quite similar locklosses also this morning before maintenance, so it's not super likely that this will magically fix things, but it probably won't hurt.

I saved all the offsets I could find in SDF, but I didn't screenshot them.  I did capture a screenshot of the OMC DCPDs' offsets, since those seemed like they had in place quite round numbers, not values that would have been put in with a script.

I ran /opt/rtcds/userapps/release/isc/h1/scripts/dark_offsets/dark_offsets_exe.py after TJ had taken the IMC to offline, and the script did all the heavy lifting (measured and wrote values).

As per WP 11826 I reconfigured the firewall on h1daqgds0 and h1daqgds1 to allow transfer of the live CDS data to a new nds2 node in the LDAS cluster.  This was a simple update of the config with no impact on the daqd or existing data streams.

When the new nds2 node (nds-4) is ready I will enable the data stream.  There is no addition configuration required for this on the h1daqgds machines.

Jennie, TJ, Camilla

The Operator team has been seeing more locklosses at state 557 TRANSISITON _FROM_ETMX, more so when the wind is high. Times: 1397719797; 1397722896; 1397725803; 1397273349, 1397915077

Last night we had a lockloss from 558 LOWNOISE_ESD_ETMX with a 9Hz ITMX L3 oscillation, see attached. Compare to a successful transition (still has glitch).

Note that there is a glitch ~30s before state 558 in both cases.  H1:SUS-ETMX_L3_DRIVEALIGN_L2L and _L3_LOCK_L filter changes happens here. Are ramp times finished before these changes?

The timing of the glitch is 2 m 55 seconds after we get to state 557, this is the same time as in the 4 of the last 6  state 557 locklosses.

I've added a test to Verbal Alarms that will notify when the PSL chiller is "alarming," which most likely means the chiller's water level is low. This is being added on top of the already existing DIAG_MAIN notification for the same event. When the H1:PSL-LASER_CHILLER1_ALARM channel flips to 1, Verbal Alarms will say "Check PSL chiller," then wait four hours before saying it again if the alarm channel has flipped again since then. The reasoning behind this is that the chiller can run for a while (unsure of exactly how long, but past examples have shown the alarm running for up to almost 24 hours) before it will actually turn off, taking the NPRO and amplifiers down with it.

If operators receive this warning, they can relay the message to anyone on the PSL team or refill the reservoir themselves if on a weekend, making sure to note how much water was added.

EY dust monitor pump 0715002111 failed. Replaced with 11001311. The pump that failed was left at the end station to cool down. I will retreive and fix the pump next Tuesday.

Ex and corner station pumps running smoothly and within temp range.

Nick, Robert, Mike

In December of 2023, Atlas Agro, who plans on building a fertilizer production plan that is net carbon neutral on DOE land, made some test seismic noise by dropping a 1600-lb block into an excavated hole. Atlas Agro made nearly 100 test drops. I was asked to see if any of the seismic noise appeared on any of our seismometers. I have been working with Robert Schofield, and have not seen any of the seismic noise. I have attached a DCC link that contains a PDF with some of the time series of the drops and includes links to the rest of the time series.

https://dcc.ligo.org/LIGO-T2400124

Jennie W, Camilla

Today Camilla and were trying to move around the IM1 and IM2 mirrors in turn and try and clip the beam going through the Faraday Isolator to see how centred we are.

Details in a log from Camilla.

As part of this we need to make a move with either IM1 or IM2 and bring the beam back onto the centre of IM4 TRANS with IM3 pitch or yaw moves.

Jenne had the idea we could use a DC centering loop to speed this up by moving IM3 for us.

We used the DC7 centering path after trending the outputs of this back to make sure it was not used during locking or in nominal low noise - checked back at least 70 days and the outputs were zero.

I checked the DC1 centering loop filters and copied over the three that are on during lock for both yaw and pitch into the DC7 foton files, putting all three filters in place of the FM2 module (previously called OFF and having a value of gain(0)) in ASC.adl DC7_P and DC7_Y.

Then we altered the ASC input and output matrices so that DC7 takes IM4 TRANS QPD as an input and moves IM3 to centre on this.

We tried different gain up to +/-50 in the yaw loop but never managed to make the yaw of IM4 TRANS converge to zero.

After talking to Sheila we realised that the DC1 loop servo will not match the plant of the IM3 suspension so to use this for IM moves we would have to put more thought into the filters.

NB: My changing and saving of the ASC.txt foton file led to some confusion with Dave B. et al. who were trying to track down a rounding error in the DHARD filter in the same fofon file. But this has been resolved.

There was an increase in space temperature at End X, which is evidenced in the trend. This increase was due to our disabling the fans for quarterly greasing. 

Initially, the air handlers were configured to raise their supply air set points when in Start-Up mode; this mode occurs whenever the program loses fan status, such as during unoccupied periods, so that upon start-up, the system is not overloaded trying to pull the supply air temperature down as fast as possible. Since the only time the fans are theoretically off is when we grease them, which only takes thirty seconds, there is no need to gradually reduce supply air temperature.

I have changed the Start-Up supply air temperatures at EX and EY from 61 to 55, which is roughly the supply air temperature the air handlers normally produce. This should reduce any space temperature increases after quarterly greasing tasks. 

Karmeng, Camilla. H1:SQZ-SHG_FIBR_REJECTED_DC_POWERMON power was high. Adjusted the 1/2 and 1/4 waveplate to reduce rejected power from 0.3 to 0.03. Instruction

During yesterday's (4/16) maintenance period, a Norco tech came to the site to inspect the 8 LN2 dewars that feed the cryopumps. Inspection report will be posted to Q2000008 once received.

The vacuum jacket pressures were also measured during inspection: