Jeff wrote the more detailed alog of the problems with the interferometer and how we tried to track down the issue, but short version, I installed a bad blend filter that has been causing an offset to accumulate in the ISI RZ locations for all of the HAMs, except HAM8. The accumulations take about 2 weeks to reach 1 urad levels. I designed the filter for HAM8 and it has been running there with no problems since last year, so I installed on HAM4-5 on Dec 11 last year, and March 15 on HAM2-3 this year. Unfortunately, there are some issues with the foton implementation, either because I missed some cleanup in the filter or because I got bit by some issue with writing the filters from Matlab to the foton file. Attached screenshot shows some of the zpk coefficients from Foton. There are 3 poles at, or near 0hz, and there are two negative zeros at 10^-6 hz. I certainly didn't use those in my design in matlab, but I guess I missed a step or two in cleaning up the filters before install. I think I've got a fix for that now, but will wait for Tuesday to try. They shouldn't accumulate much offset over the long weekend.

I've also added a test to DIAG_MAIN, to warn if the errant integration reaches 1urad on any of the HAM chambers. This should take 2 weeks or more at the current rate, but if any of the HAMs reach 1urad offset, DIAG_MAIN will report:

"ISI {} RZ CPS residual bad"

The test in DIAG_MAIN is:

@SYSDIAG.register_test
def HAM_RZ_RESIDUAL():
    """HAM RZ residual test

    """
   
    chambers = ['HAM2', 'HAM3','HAM4', 'HAM5', 'HAM6', 'HAM7','HAM8']
    bscs = chambers
    tots = []
    for chamber in bscs:
        if ezca['ISI-' + chamber +'_CPS_RZ_RESIDUALMON'] > 1000:
            tots.append(chamber)
    if tots:
        yield "ISI {} RZ CPS  residual bad".format(tots)

The H1ALSEX_SDF_TABLE Changes seemed to be hand made. I checked them against the Safe.snap and the values are indeed different. So I will accept them as shown.

The H1:SQZ-OPO_TEC_SETTEMP was accepted upon request from Vicki.

The Camera Diffs went away on their own.

(Jordan V., Gerardo M., Janos C., Travis S.)

Late entry, systems ran on May 23rd, during the maintenance period.

Corner Station Dry Air Skid
We ran the vent/purge supply for the Corner station to test dew point and take particulate samples, basically a functionality test.
No issues encountered during start-up and during the period the compressor operated.  The dryer tower system ran good, no surprises.
The entire system was allowed to come up to nominal operation, while the system was on, we opened a butterfly valve located far from the dryer system, next to HAM3 to be exact, this action was done to let the purge air dry the line.
We took a couple of samples with the particle counter, after a couple of measurements were taken with the dew point meter, see final results below.  After measurements and samples were done system was shut off.
Dew point: -42.0 degrees C. this is good.
Particle counts: 0.3 and 0.5 micron size they were below 10 counts.

Y-End Station Dry Air Skid
We ran the vent/purge supply for the Y-End station to test particulate and dew point, basically a functionality test.
No issues encountered during start-up and during running compressor.  The dryer tower ran and did the switching of towers.
We took a couple of samples with the particle counter and with the dew point meter, see final results below.  After measurements were done system was shut off.
Dew point: -41.3 degrees C. very good number.
Particle counts: 0.3 and 0.5 micron size <10.

X-End Station Dry Air Skid
No results.  We need to run the same functionality test next week for this unit.

On a side note we purged the 2B turbo header using ultra zero bottled dry air, we connected the bottle to the header near but past the XBM turbo pump connection, system flushed for more than 30 minutes.

Quick summary: We've been able to get to DRMI_TO_POP and can now engage most of the DRMI ASC loops. Jim and Jeff are currently looking into an issue with the HAM yaw motion that might help explain the optic moves that we've had to make.

• PRMI will lock breifly but then quickly lose lock. This was from the ASC PRC1 gain going from 12 to 5, we bumped this up from 5 to 8 in lscparams for now. Might need to revert later.
• Tried walking PR3 in the direction that it has been moving (in the -yaw) and compensate with PR2, this didn't seem to help. We moved suspensions back to our best buildup position within this movement.
• We managed to lock PRMI and then DRMI with this alignment. We kept DRMI ASC off because the error signals were not well centered, and attempts to center were not sucessful. Most noteably INP1 and PRC2, but others were off as well. We decided to try moving past DRMI with only the MICH ASC loop engaged. We then made it up to DRMI_ON_POP, so past a CARM reduction and technically made it onto POP @1f.
• We've now repeated this a few times with little or no alignment changes, so we will step through the state by hand to find if there is an exact spot that it is failing.
• Jim had a look at the HAM CPS residuals and noticed that they have been exponentially driving away for the last ~45 days (plots attached). This might line up with the adjustments we've had to make in the input arm. Jim and Jeff are currently investigating.

During our locking troubles over the last few days, the LVEA temperatures have been in flux from fan flow changes. The temperatures have definitely stabilized over the last 20 hours, but they are higher than before these tests, zone 4 in particular. Just to rule out any rubbing we ran Jenne's ipynb that checks the top mass OSEMs with a reference time (alog). I tried a few different reference times with the same results - I didn't see any evidence of rubbing on these plots. Plots attached.

Edit: The pdf didn't have all the pages and I've struggled to get them all in and have the alog accept the upload. The currrent attachment has sort of visible plots, but if you're really interested it might be better to go open the notebook up yourself with these already saved.

As suggested by Dan Brown, I checked if the HWS could see any ITM beam spots movement during the May 10th/11th temperature excursion alog 69525. I could see nothing.

The attached plots of ITMX and ITMY show (L to R) the two locked before temperature changes and the two locks during temperature change. ndscope attached. HWS ref at at 76W input power compared to show time is 5 minutes after. Lock during the change was 50 minutes long. Videos didn't show anything either.

Continuing w/ PR3 move recovery.  Still having issues with PRMI.

Ran a Manual Alignment 2-3 times after TJ did one.  

• Once again INPUT ALIGN was finicky.  There were flashes on the order of 0.5-0.8 for TR_X, but no locks.  I tried PR2 yaw moves to try and improve alignment/flashes, but this did not help.  I also tried small moves w/ IM4, but that had no effect.
• Quit INPUT ALIGN since it was not working out.  
• Continued with other alignment steps (except green arms since they are fine).
• SRC Align continued to not work, so I did an SR2_ALIGN & OFFLOAD and then aligned SRM by hand.
• Then took a look at PRMI, but it was lifeless again (on ndscope) and misaligned on yaw via AS_AIR camera.
• Currently taking INPUT_ALIGN to ACQUIRE_XARM_IR
  • Sheila had some checks I could do here:  watch AS_AIR at this step to get an idea of how input alignment is misaligned (so would have idea of which way to point PR2)
  • Also could double the gain (i.e. from 0.04 to 0.08) for H1:LSC-XARM_GAIN to help Xarm lock (and restore gain after optimizing alignment)
  • ......But with Sheila on Teamspeak, she noticed a dark offset which was a bit off.  She fixed this by-hand and then TR_X immediately locked.  She touched up some of the alignment and then we let ALIGN IFO take over.  
  • Finally INPUT ALIGN completed successfully!
• Now going to rerun PRC ALIGN, MICH BRIGHT and attempt SRC (if latter doesn't work, will skip).
• Then try PRMI again and check for improvements.

TITLE: 05/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY: We are still trying to get our comm/diff beatnotes and PR3 alignment understood. Sheila and Ryan S went on table this morning (alog69907) and maximized our beatnotes with our old PR3 position (Yaw slider 148.5). We struggled to get DRMI locked well after that. Sheila noticed that our green qpd offsets and ITM camera offsets were set back to the old values, incorrect values, so they might have not been saved in SDF and then reverted. Sheila and Ryan made a second trip to the table and retouched the alignment with the correct qpd offsets, camera offsets, PR3 settings, and aligned green arms. Trying to lock again but it looks like we need another initial alignment, running that now.
LOG:

TITLE: 05/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 8mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

TJ was at early steps of locking (noticed some odd offset sweeps for COMM/DIFF while FINDING IR).  PRMI does not look very good sadly and he's moving to CHECK MICH FRINGES.

Since the table work this morning we have been struggling to get past DRMI or engage DRMI ASC. Catching PRMI can even prove to be challenging and I've been skipping over PRMI ASC for now. We will next try to get the ASC to work and zero the error signals if we can get back to a stable PRMI.

Sheila, Ryan S, TJ

This morning we continued the ISCT1 and PR3 alignment from 69903 , 69897 ,  69857. 

TJ reverted the X arm green references to what they were yesterday morning, seen snapshot here, and re-ran intial alignment of the X arm.  Ryan S and I went to ISCT1 and saw that the beatnotes weren't well aligned, but the whole table alignment was better than what TJ and I found yesterday.  We toggled PR3 between 151.6 and 148.5 yaw slider, and saw that the ISCT1 alignment was sort of close to OK for both of these, such that beams were clearing their appertures and making it to the diodes they should have without clipping. 

This means that after our table realignment yesterday, the table was in a sort of OK alignment (beams not totally missing their optics) for 4 configurations: both of the green arm alignment set points and both the PR3 pointings.  This means that the table alignment that TJ and I did yesterday fixed some gross misalignment that wasn't due to either of those things, (it has been a long time previously since the alignment on table was fixed up). 

From the control room TJ, Ryan and I, with PR3 set to 148.5 yaw slider (setting for locks yesterday), adjusted PR2 to lock the X arm in IR, ran the WFS for INPUT align, and did a MICH bright alignment.  Ryan and I returned to the table and moved the bottom periscope mirror to improve the alignment of the Y beam onto the trans diode, which allowed TJ to lock the Y arm in green and run inital alignment for that.  In the meantime, Ryan and I adjusted the position of the X trans diode, and improved the COMM beatnote using the beatnote BS and the one upstream of it in the X arm path.  We eneded up with a COMM beatnote of 6dBm, which drifted down later when TJ re-ran inital alignment for the X arm and is now at -1dBm. 

After TJ finished the Y arm green inital alignment, and a MICH bright alignment, Ryan and I realinged the DIFF beatnote, which is now -7dBm.  TJ then completed the intial alignment steps we had skipped (PRC and SRC) and is now trying to lock DRMI.

TITLE: 05/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 8mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY: The IFO is still down since the PR3 and ISCT1 work yesterday. It's sounding like these changes will need to be reverted this morning.

TITLE: 05/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY:

Majority of the shift was focused on attempting to get H1 to lock post ISCT1 table work (worked with Sheila a little before she left for home and then with her over teamspeak).  Several Full & Manual alignments were attempted with various changes made to PR2 (and a little with PR3).  Unfortunately, all the alignments led to poor locking of anything past Green Arms.  

• Corey had difficulty locking PRMI after the changes to PR3 and ISCT1 alignment we made earlier today (69897). 
• Right now Corey is going to retry initial alignment, hopefully input align will go more smoothly than it did earlier in the evening and allow Corey to lock PRMI.
• If PRMI will still not lock after this, we can move PR3 back towards it's position of this morning. 
  • Right now PR3 is at 151.6 on the yaw slider, this morning it was at 148.5 yaw. 
  • With the current X arm green references and ISCT1 alignment, it seems that we can move PR3 to 150.2 on the yaw slider to have a beatnote of -16dBm.  This should be OK for locking, but means that we have clipping on the way to the trans PD.  To make our automation work with less light on the PD, we need to renormalize it both for Beckhoff and the front end versions of this channel:
    • for Beckhoff we change H1:ALS-C_TRX_A_DC_NOMINAL from 0.054 to 0.02 (can be done through ezcawrite, or from ALS overview, X PDH screen (grey hidden button), X arm green transmission link on the bottom of the screen).
    • the front end version is H1:ALS-C_TRX_A_LF_GAIN, ALS corner overview, TRX_A_LF orange button in the greenpath box lower left.  We set this to 1.35 earlier today, it might need to be more like 4 or 5 to bring the normalized channel to 1 if PR3 yaw is at 150.2 yaw. 
  • After moving PR3 to 150.2 yaw slider and renormalizing, Corey or Ryan would need to redo the inital alignment starting from INPUT ALIGN.Then they can try to lock PRMI.
• If neither oif those work, it is time to think of reverting what we did this afternoon, including the green references for the X arm, the PR3 move, and the alignment on ISCT1.  This is a project probably best left for the morning.

We are still having difficulties with SRY locking, but I haven't looked into what is going on.  If SRY is not locking in an inital alignment step, it is OK to run SR2 ALIGN offload, and move SRM by hand to increase the SRY flashes. 

Editing to add:

Input align wasn't working well because of an apparent dark offset in POP A 45I.  I adjusted the dark offset from -68.9 to -72 to get the signal without flashes to be near zero.  This must be reset by one of the init align guardian states.  This problem has been seen before and TJ attempted to fix it here, but something is still not working.

2nd edit:  We weren't able to get above 30 counts on POP18 + POP90 in PRMI with PR3 at 151.6 yaw slider, so we moved it to 150.2 with the renormalizations above and re did initial alignment of the IR cavities.  Input align wouldn't lock but the flashes were at 1, so we moved on.  SRY has worked twice in a row.  With this lower beatnote we've had an ALS lockloss, I think that if a few more tries at this alignment aren't fruitful it would make sense to cancel the owl and go to the table in the morning.

After the table work, ran an Initial Alignment (it was a bit rough in that INPUT_ALIGN had trouble locking the X-arm.  Restrored PR2 alignment to where it was during the last lock, but it was still tough.  However, INPUT ALIGN finally did complete and it looked as though the entire alignment completed.  

Unfortunately, DRMI did not look good and PRMI also looked misaligned (there were short 1-2 second locks, but they are very low and so far I have not been fast enough to touch up the PRMI alignment to let ASC take over).

TJ, Sheila, Betsy, Jim 

Since we have had increasing drift in the PR3 yaw alignment needed to get ALS beatnotes good enough to lock, we realigned the beat notes on ISCT1 this afternoon.  First we set the green x arm references as we found them yesterday while locked with 2W and the spots centered on the optics (69857).  Screenshot attached of accepting these new references in SDF.  

TJ then ran an intial alignment through the MICH alignment step, we saw that COMM beatnote was -18dBm, TJ adjusted PR3 yaw to 147.2urad to increase the COMM beatnote to -10dBm; 2 weeks ago PR3 yaw was at 151.6 urad so this is an pparnt drift of 4.4 urad.  The osems and optical lever do not indicate that PR3 has had any drift like this.  

TJ and I went to ISCT1, looked at the beampaths which were mostly reasonably centered with PR3 at 147.2urad, then asked Jim to move it to 151.6 to restore PR3 to it's alignment of 2 weeks ago.  Then the Y arm beam was hitting the X arm side of the prism and being directed into the X arm path.  We adjusted mostly yaw and a little pitch on the bottom periscope mirror, (layout here), as well as the yaw of the 1 inch BS that splits the X beam between the trans PD path and the beatnote paths, to get both X and Y beams roughly aligned through the paths to the comm and diff beamsplitters. 

From there we manually aligned the COMM beatnote, with an IR card then the scope, and got a beatnote strength of 7-8 dBm, for DIFF we have -8.2dBm.  The arm transmission diodes were lower than previously, so Betsy and I went back to the table and checked that the beams were centered on the trans PDs, which didn't increase the power.  Corey and I then re-normalized the trans diodes, by adding a gain of 1.16 to the Y channel and 1.35 to the X channel.  The changes in screenshots on this alog will need to be accepted in OBSERVE. 

TITLE: 05/24 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:

Lock#1

Couldn't lock PRMI, lockloss

Lock#2

Went right into an initial alignement, OM1 & OM2 were saturated during PRC in initial alignment

DRMI's lock didn't seem great on the buildups but the spot looked ok

OMC locked first try on its own

NLN @ 08:44, 30 ASC SDF diffs but they were all for the camera servos, I waited for ADS to converge for the camera servos to turn on (~ 16mins) which cleared these diffs

Observing mode @ 09:02 while we thermalize

Out of observing at 11:05UTC for a new FF filter test/measurement and then a calibration suite, new FF filter applied at 11:07UTC

I used Coreys template (/ligo/home/corey.gray/Templates/dtt/DARM_05232023.xml) which I had to enable "read data from tape" for it to run. Measurement started at 11:10UTC, finished at 11:12UTC. My DTT session then immediately glitched and crashed before I could save it, great... I restarted the measurement at 11:15UTC, finished at 11:16UTC. It wouldn't let me save it (error: Unable to open output file), but I added it as ref 27 on the previously mentioned xml file in Coreys directory but this might have not saved from that issue. I grabbed a screenshot of it.

I switched back to the old filters to take the calibration suite at 11:21UTC, I wasn't sure if we wanted them on or not for this, apologies if we did want them on.

Lockloss @ 11:25, possibly from PI29, but on NUC25s scope guardian appeared to be successfully damping it? It was tapering down when when the DCPDs saturated and we lost lock, it also coincided with a ground motion spike from that 5.5 from NZ. I then stepped down the EX ring heaters to 1.2 using the console commands Sheila provided in her alog.

Lock#3:

Yarms power was drastically lower after the lockloss and looked clipped on the camera, increase flashes ran twice and wasn't able to get it above 50%, I stepped in and still wasn't able to even get it to 80%. I gave guardian another shot after this and increase flashes ran another 2 times and was not able to get it, I tried again to lock it and was unsuccesful. Lockloss at LOCKING_ALS after some more rounds of adjusting and trying to lock.

Lock#4-11

ALS locklosses

Lock#12

Beatnotes aren't great, -19 & -20, I'm starting another initial alignment. Lots of SRM saturations during SRC align, trending SRMs OSEMs there doesn't appear to be any unusual motion in the past 20 hours.

After a suggestion from Betsy and Jenne, I check PR3 and it seem to have drifted a bit, I moved PR3 in yaw 0.8 microradians negatively which increased the COMM beatnote. I was able to lock ALS after this but got no flashes on PRMI so I started another initial alignment but it didn't do the SRC align correctly again, TJs going back to try and fix this.

Handing off to TJ

LOG:

We lost lock from what seems to have been PI29 at 11:25 UTC (I stepped down the EX ring heaters to 1.2 following Sheilas alog after that) and I haven't been able to lock ALS since, green arms was a big struggle especially Yarm. After a bunch of rounds of adjusting, letting increase flashes run, and a few ALS locklosses I started another initial alignment at 12:32UTC.

TITLE: 05/23 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
SHIFT SUMMARY:

Frustrating shift sandwiched with tough locking at early states of ISC_LOCK:  Mainly LOCKING_ALS.  Seemed to get better, but after MANY attempts and atleast an hour!

Had one lock with some observing time, but it was another shortlived lock (~2hrs).

LOG:

• PCalx/y Test while at OMC WHITENING, in prep for OMC Refl (Jennie)
• 0018 NOMINAL LOW NOISE
  • OMC Refl measurement completed (jennie)
• 0020 Transitioned OBSERVATORY MODE to COMMISSIONING (from Preventative Maintenance)
• 0033 LOCKLOSS due to an errant sdf revert of Feed Forward filters
• Early LOCKS:
  • Repeated locklosses at around LOCKING ALS & FIND IR.  No errors or DIAG MAIN notifications gleaned any hints at the reasoning.
  • 0131:  Paused at LOCKING ALS and FIND IR but did not seem to help, and went straight for DRMI & made fairly randomly it through.
  • At DRMI acquisition,  looked misaligned.  Went for PRMI, but it also looked misaligned in yaw from the short locks---tried moving PRM in yaw to no avail.
  • CHECK MICH FRINGES run 
  • PRMI then locked with better alignment...but DRMI did not lock (even as I tweaked SRM)
  • PRMI automatically run and then DRMI finally locked quick!
  • ENGAGE ASC FOR FULL IFO took quite a bit of time due to ADS being off quite a bit (mainly YAW3).
  • 0240 NOMINAL LOW NOISE (but waiting on Camera Servo)
  • 0305-0311 Vicky checking on squeezer.
  • 0311 OBSERVING
• 0419 ITMx, ITMy, BS saturation on verbal alarm (but didn't see anything on DARM).
• 0445 LOCKLOSS:  Vicky confirmed this was NOT a PI Mode lockloss.
  • Issue of recurring locklosses at LOCKING_ALS happen again.
• 0511-0528 Initial Alignment
• 0529 Diag Reset on SEIH16 since it had a RED ADC.
• LOCKING NOTES (post alignment)
  • Continue LOCKING_ALS locklosses
  • Paused at LOCKING_ARMS_GREEN (doesn't seem to help---as soon as I go to LOCKING_ALS this is when lockloss occurs).
  • Not consistent, but locklosses happen after ALS_DIFF is at LOCKED & during different states of ALS_COMM (such as HANDOFF_PART2 3x in a row, but it's also happened at Part3 and eralier at FINE_TUNE_IR).
  • After an hr+ of attempts of getting past LOCKING_ALS/FIND_IR, finally managed to get to ACQUIRE_PRMI (I went specifically for this state because I was sure the alignment was not going to be great after repeated failed attempts of getting past LOCKING_ALS.  PRMI was ugly, but was able to tweak PRM to inch it up a little, but it dropped.  Had several short locks and wanted to get ASC to take over but then there was a lockloss.
  • This is how locking has been for my two attempts at locking for the shift.  For the lock before my shift, TJ zoomed right past this on his first attempt (vs the dozens of attempts I needed for both of my attempts.
  • Also for both of my locking attempts tonight, it made it past LOCKING_ALS eventually on its own---which led me to wonder if there could be a thermal effect?
  • Have since made it to PRMI once again (but had a lockloss when transitioning to DRMI)....this happened during the handoff to Ryan C.

Sheila, TJ, Betsy, Keita, Jeff 

Over the last 9 days we have moved the PR3 yaw slider by 2 urad (from 151.6 to 149.3) to maintain the ALS beatnotes on ISCT1, to allow locking.  Today we had planned to move PR3 back and realign on the table, but when I set PR3 back TJ saw that the beams were clipped arriving on the table, and not reaching the trans PDs well enough to trigger and lock the arms in green.  The beam was clipping on the prism, so fixing this on the table would have meant moving the periscope mirrors.  We decided to first double check that the green camera for the X arm had not drifted.  

Instead of realigning on the table, we decided to check the camera reference.  TJ skipped shuttering ALS as we relocked, let ADS converge at 2W, and Keita and I ran the green offset servo (reminder, green reference instructions are here).  The attached screenshot shows the QPD offsets and camera offsets that we would have gotten.  These are not large changes to the references, so it doesn't seem that the problem is something like a drift of the camera. We reverted these changes to the green references. 

A screenshot of the camera is attached, which does show the beam is not well centered on this camera. 

We are returning to locking now, but since this has been causing locking troubles and seems to be getting worse over time, we will plan to return to re-align the table next time we are relockign during the day.