TITLE: 06/01 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 134Mpc
SHIFT SUMMARY:

Handing off to Camilla

• Observing since 18:58UTC
• Event candidate

Lock#1:

The failed Coildriver was replaced around 17:00UTC (FRS ticket) and I began relocking, I tried quickly to see if it could get past DRMI but no luck so I went into an initial alignment. During the IA TJ adjusted SRM during SRC align as it was not well aligned.

NLN reaquired @ 18:45 UTC with no interventions. In observing @ 18:58UTC

EX saturation @ 19:33, 20:51, 20:52, 21:07 UTC

I cautiously tried to find a better setting for IY5/6, I first tried stepping up the gain to -0.04 and it brought it down more than twice as quickly (~30%/hour)

WP 11238
FRS 28121

EX PUM Coil Driver Chassis pulled by R. McCarthy for troubleshooting. Differences noted between spare and installed unit:

1. EX unit has new noise monitor board installed, spare unit has original noise monitor board
2. EX unit has ECR E2100204 implemented, spare unit did not

Replaced failed capacitor on EX unit (Channel 2, C24). Capacitor was shorting +18V rail to ground. Ran transfer functions on all channels in both Low Pass and Acquire mode. Using CH1 as a baseline, all channels gave same result. Unit reinstalled.

Spare unit will be modified per ECR 2100204.

PUM EX- S1000343
PUM Spare - S1102649

F. Clara, R. McCarthy

Thu Jun 01 10:07:58 2023 INFO: Fill completed in 7min 57secs

Jordan confirmed a good fill curbside.

WP11227 FRS27187

Jonathan, Fil, EJ, Erik, Dave:

We have had zero ADC1-TIM errors on h1seih16 since Erik moved all the cards from the second Adnaco backplane over to the unused third backplane 48 hours ago. Previously we had at least one error per day, so it looks like this problem was with the computer to Adnado link and presented itself as a timing issue with the first ADC present on the backplane, ADC1 in this case.

Note this most probably means that the original ADC1 which was removed Tue 23may2023 is most probably a good ADC. SN=110124-06

WP and FRS have been closed.

TITLE: 06/01 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

Taking over from Ibrahim

• Currently waiting on Fil to replace the failed coil driver on ETMX
• CDS looks good, the winds have died down since last night, no alarms

TITLE: 06/01 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY:

1. Lockloss at 12:44 UTC due to EX ISI movement it seems. The PUM, ISI Stage 1 and 2 WDs all tripped about 10 seconds prior 
   1. ISI WDs untripped
   2. PUM WD is not resetting and is showing a big “Rocker Switch Death” sign
2. Lock (Attempt)  2 - “ROCKER SWITCH” Edition: 
   1. Put IFO in down until the Rocker Switch issue is fixed.
   2. Texted LHO Ctrl Rm and Jenne came on and helped get the ISI WDs untripped and discuss the not-so-normal rocker switch issue.
   3. There were no earthquakes, wind or other potential explanations for both the ISI and the PUM WD trip - Per Jenne’s discussion, this suggests an electronics/hardware issue:
      1. No Signals from OSEMs
      2. Big red ROCKER SWITCH DEATH seems to be an “and” condition of the OSEM inputs not working (per Jenne’s help finding the right alog)
      3. PUM Coil WD RESET button not working
      4. Wiki’s helpfulness was limited 
   4. Richard McCarthy is going to EX to investigate the coil drivers.
      1. As Richard is investigating at EX, I see the Rocker Switch Flickering much more frequently.
      2. Secondary ISI WD trip due to Richard arriving at EX - will keep tripped until investigation is over.
   5. Upon Richard checking, it seems this was a bad/blown coil driver in the EX PUM. It will need replacement. Putting ops mode in maintenance.
      1. Richard reports a 15V rail

EX PUM Coil Driver Failure Story Summary and Findings

See above for more detailed LOG of events.

At 12:44 UTC the ISI and PUM Watch Dogs tripped seemingly out-of-nowhere.

From the above details, we can safely conclude that the PUM Coil Driver failure is what caused both ISI and PUM watchdogs to trip. It also explains why the PUM WD button “doesn’t work” - it’s that the WD is just being re-tripped as the driver continues to fail and flail. Richard called Fil and he’s on the way to do the needed maintenance. IFO is in down until and OPS mode is in “corrective maintenance” until the issue is resolved.

LOG:                                                                                                            

1. Lockloss at 8:41 UTC for unknown reasons (DRMI unlocked, though not much ground motion or wind).
2. Lock 1: It was a fully automated lock and until violin damping, was the most stable/easiest one I’ve seen yet! It would’ve been a sub-40 minute lock if not for IY Violin modes.
   1. Increase flashes for both ALS X and Y.
   2. DRMI Locked without going to PRMI
   3. Reached OMC_Whitening (state before NLN) at 9:34 UTC but it took an additional 37 minutes of just damping IY modes since those were heavily rung-up. Didn’t touch anything though since they were all going down as expected.
   4. Reached NLN at 10:11 UTC.
3. Other: DARM FOM crashed when we lost lock and since restarting, it is glitching every so often.

IFO STATUS: IFO is at NLN and in OBSERVING as of 10:13 UTC

TITLE: 06/01 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 131Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 9mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.10 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 7:01 UTC

TITLE: 06/01 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 132Mpc
SHIFT SUMMARY: One lockloss from the wind, one from an unknown cause (length glitch beforehand), just got back to Observing.
LOG:

Closes FAMIS 25068 and  FAMIS 25066.

All plots are attached. Only seeing some slow V_eff drifts. This agrees with what Ryan and Rahul are seeing in the Oplev charge measurements (69432 69252). We can compare them with the script Ryan C made, instructions in 69252, see attached plot. Both in-lock and op-lev trends show V_eff moving away from zero, we wonder if a sign is incorrect in ETMX.

Two FAMIS tasks closed as Austin was unable to run the 16th May measurements analysis as the log file hasn't been clearing correctly. I've corrected the logfiles and edited the RUN_ESD_EXC.py code to correctly clear the logfile, now using logfile.clear() rather than logfile = [] in (69233). 

These charge measurements have been causing a lockloss just before Tuesday maintenance (69437) so I've lengthened the final ramp time back from 20 seconds to 60 seconds (69234). 

We were observing at 135+MPc for the end of a 5h30 lock until a lockloss at 01:19UTC, 1369617587. We had wind gusts of 30+mph and a little ground motion from a small earthquake that I think caused this lockloss. Some 2-4Hz wiggles beforehand.

Currently at state MOVE_SPOTS after an unsuccessful initial alignment. INIT_ALIGN SRC alignment wasn't working so I skipped it and touched SRM by hand in DRMI locking.

I tried a few permutations of turning on the retrained cleaning that I described in alogs 70043 and 70023.  In the end, I'm just leaving on the updated jitter subtraction.  The LSC subtraction worked well, but then we updated some parameters (see Elenna's alog 70052), and it now needs to be retrained. The laser noise subtraction is quite confusing, and so is not on.

The first attachment shows all 3 (LSC, jitter, laser noise) updated subtractions on.  You can see that the low freqs (below ~500 Hz) shows improvement, as does very high freqs (above 5 kHz).  But, the middle-high freqs (500 Hz to 5 kHz) has injected noise.  The green trace is the contribution from the Laser noise subtraction, and you can see that it is the dominant term above 500 Hz.  This implies that we've got something strange going on that we have a sign flip around 5 kHz somewhere. It's possible that this is due to not long enough thermalization, but I suspect that we're just missing some term or sign or filter phase somewhere.

The second attachment shows just the LSC subtraction, and it does very well.  This plot was taken with all old settings (old SRCL offset, old SRCL FF), so is just illustrative of what we should be able to achieve.

The third attachment shows just the Jitter subtraction, and it also does very well, except that I've had to put in a minus sign in the filter bank gain.  A difference between what I had been using for Jitter subtraction and what is now currencly running is that what had been in use was trained on a measured exported-from-DTT transfer function between the OAF-NOISE_EST channel and GDS-CALIB_STRAIN_CLEAN, and today I'm using the filters that should be in use in the GDS calibration pipeline. Those two things have a difference of a minus sign, which is why today I have a minus sign in the filter bank gain.

Since the SRCL FF was changed after these plots were taken, the LSC cleaning now needs to be retrained.  I'll get some data from overnight tonight, and retrain the LSC tomorrow.

The fourth attachment is the range for our 3 versions of CALIB_STRAIN from the summary page.  Around 2300 UTC, there is a marked difference between pink and blue - that's with both LSC and Jitter subtraction on, after Shiela tuned up the squeezer.  Then the last time the range is high in this plot, around 2330 UTC, only the Jitter subtraction is on, but Elenna had turned on the updated SRCL FF filter, which is why our overall pink and orange range is higher than at 2300 UTC. Hopefully once I've retrained the LSC cleaning, we'll again get blue up a little higher.

I've accepted all the associated SDFs for these in both the safe and observe snap files.

I took some recent times at 60W and 76W and made some power trend plots of the first three hours of the lock. The 60W time is from a long lock around March 29th, and the 76W time is from a long lock around April 29th. Unfortunately the time for the April 29th lock we had the POP beam diverter closed, so we are missing the sideband trend during thermalization.

During the commisoning window this afternoon we got some data for different squeezer configurations.  

Here is a summary of times:

First, a set of data at the current SRCL offset, summarized in the first screenshot, references are saved in /ligo/home/sheila.dwyer/SQZ/SQZ_vs_SRCL/SQZ_vs_SRCL_DARM_FOM.xml. 

• FDS, SRCL offset -290, SQZ phase 143.71 (configuration we have been observing with) ref 0, 31/05/23 20:57 UTC, CLF servo sign +
• No SQZ, SRCL offset -290 21:16 UTC (ref 1)   increased OPO trans to 61uW, changed OPO temp, CLF RF6 ABS went from 0.0493 to 0.0554, 
• FIS 21:25 ref 2 UTC SRCL -290 sqz ang 143.71 (we did check for a better sqz ang, but there wasn't much benefit to changing the angle here) , CLF servo sign +
• FI anti-sqz, 21:44:00 SRCL -290, sqz ang 68, CLF servo sign - 
• mid way sqz angle, 21:52:49 ref 4, SRCL -290 sqz ang 205, clf servo +

Then we tried to repeat a scan of sqz angle and srcl offsets similar to what Vicky and Elenna did here, however the impact wasn't as noticable as in that alog, perhaps because we don't have a lot of anti-sqz right now.  Second attached screenshot shows the summary of this. 

• ref 5 21:52:49 UTC, SRCL -200, sqz angle 143.71 CLF servo +
• ref 6 22:01:39, changed sqz angle to restore high frequency sqz, SRCL -200, sqz ang 113.64, clf servo +
• ref 7 22:09:53 UTC, SRCL -165 sqz ang 83.57 CLF servo +

Then a repeat of the same data set as above but taken at a SRCL offset of -165, as shown in the third attachment. 

• ref 8 22:14:45 UTC, SRCL -165 sqz angle 58.51 CLF servo - (anti-sqz)
• ref 9 22:23:16, SRCL -165 sqz angle 198.84 servo + (mid sqz)
• ref 10 22:28:48 SRCL -165, no sqz

We have recently reduced the ETMX ring heater by 0.1 W to avoid the 80 kHz PI. As a result, our buildups have increased slightly and we have been operating closer to 440 kW. We ahve also had much shorter locks than we have had in the past operating around 430 kW. Looking at the power and lock trends, our longest locks occur when the circulating power is slightly lower. This small variation in circulating power can result from slightly different input powers every lock from how the ISS second loop closes. Jenne and I think this is enough evidence to drop the PSL input power by 1W. We will now operate at 75W input from the PSL, which should bring us closer to the 430 kW operating power that works best for us.

The requested power level for NLN in lscparams is now 75W. I have loaded the ISC_LOCK, IMC_LOCK and LASER_PWR guardians accordingly.

This will be a catch-up Alog as it dates back to last week. It started with trying to adjust the airflows (per Robert S.) and the rising temps in the LVEA, so I started investigating and found a large amount of condensate on the floor of AHU-1. We (Tyler, Randy, Chris and I) cleaned up all of the water with wet/dry vacuums and then adjusted mechanical linkages on Fan 1 & 2 to achieve the desired airflows. 
At that time, it was discovered that cooling coil 4 was much warmer than the other coils. I was not sure what other adjustments had been made by others so I continued to monitor. I took Friday off but was checking coil 4 and talking with Richard throughout the day. Towards the end of the day coil 4 was still much warmer than normal and allowing warm air into the LVEA. I decided to turn Fan 4 completely off to see if I could get the temps to stabilize. This strategy helped. 

Yesterday, being maintenance day, I took the opportunity to investigate a little more into why coil 4 was not cooling. After checking functionality of all the valves, I decided to open the strainer and found that it was very full of debris. I cleaned the strainer, reinstalled and turned Fan 4 back on. The coil is now running at normal temperature and the LVEA is coming back to a stable temperature. 

Because of the rising temps last week, many of the zone temperatures were lowered to try and help with the warmer areas. These setpoints have all been returned to the original desired temps. 

A work permit will be put in soon to inspect and clean all remaining strainers on the next Tuesday maintenance period.

We've been having issues during initial alignment with the ALIGN_IFO node falsely convinced that SRY is locked when SRM is actually very misaligned. This was due to the SRCL trigger turning on too early (see H1:LSC-SRCL_TRIG_MON). Sheila and I looked at the LSC SRCL trig thresholds and decided to bump these up to ON:0.3 OFF:0.2 (last two changes of this - alog54799 and alog63678). I also added a trigger delay of 0.5s since it seems to flash above this threshold even at times that we are very misaligned. This trigger looks at the POP A DC signal, so we aren't sure why the amount of light has changed on this PD. These settings worked for the one time we tried it with a well aligned SRM, so we should keep an eye on it the next few times we do alignments.

This second alog63678 reminds future us that we should recommission SRY to use the AS WFS rather than the REFL WFS. I'll remind future, future us the same thing.

TITLE: 05/24 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:

Lock#1

Couldn't lock PRMI, lockloss

Lock#2

Went right into an initial alignement, OM1 & OM2 were saturated during PRC in initial alignment

DRMI's lock didn't seem great on the buildups but the spot looked ok

OMC locked first try on its own

NLN @ 08:44, 30 ASC SDF diffs but they were all for the camera servos, I waited for ADS to converge for the camera servos to turn on (~ 16mins) which cleared these diffs

Observing mode @ 09:02 while we thermalize

Out of observing at 11:05UTC for a new FF filter test/measurement and then a calibration suite, new FF filter applied at 11:07UTC

I used Coreys template (/ligo/home/corey.gray/Templates/dtt/DARM_05232023.xml) which I had to enable "read data from tape" for it to run. Measurement started at 11:10UTC, finished at 11:12UTC. My DTT session then immediately glitched and crashed before I could save it, great... I restarted the measurement at 11:15UTC, finished at 11:16UTC. It wouldn't let me save it (error: Unable to open output file), but I added it as ref 27 on the previously mentioned xml file in Coreys directory but this might have not saved from that issue. I grabbed a screenshot of it.

I switched back to the old filters to take the calibration suite at 11:21UTC, I wasn't sure if we wanted them on or not for this, apologies if we did want them on.

Lockloss @ 11:25, possibly from PI29, but on NUC25s scope guardian appeared to be successfully damping it? It was tapering down when when the DCPDs saturated and we lost lock, it also coincided with a ground motion spike from that 5.5 from NZ. I then stepped down the EX ring heaters to 1.2 using the console commands Sheila provided in her alog.

Lock#3:

Yarms power was drastically lower after the lockloss and looked clipped on the camera, increase flashes ran twice and wasn't able to get it above 50%, I stepped in and still wasn't able to even get it to 80%. I gave guardian another shot after this and increase flashes ran another 2 times and was not able to get it, I tried again to lock it and was unsuccesful. Lockloss at LOCKING_ALS after some more rounds of adjusting and trying to lock.

Lock#4-11

ALS locklosses

Lock#12

Beatnotes aren't great, -19 & -20, I'm starting another initial alignment. Lots of SRM saturations during SRC align, trending SRMs OSEMs there doesn't appear to be any unusual motion in the past 20 hours.

After a suggestion from Betsy and Jenne, I check PR3 and it seem to have drifted a bit, I moved PR3 in yaw 0.8 microradians negatively which increased the COMM beatnote. I was able to lock ALS after this but got no flashes on PRMI so I started another initial alignment but it didn't do the SRC align correctly again, TJs going back to try and fix this.

Handing off to TJ

LOG:

Although we did not get any inlock charge measurements today, we did get a new value for ETMY last week so I reran my compare code after making some updates to it.

If anyone else wants to use it, its located at: /ligo/home/ryan.crouch/Desktop/Charge_measurements/compare.py and you should just be able to run it as a regular python script. The only things one might need to modify in the code would probably be the plotting range at the top if you want to change it from the 6months its set to currently.

IE:

python3 /ligo/home/ryan.crouch/Desktop/Charge_measurements/compare.py