I have found a new setting for ITMY mode 07 since the nominal settings was making it grow (past couple of locks). Given below are the settings which works - see ndscope attached  (old settings shows the mode is growing and the new one drops quickly) below,

New: FM1+FM10, Gain 0.1

Old: FM1+FM2+FM10, Gain 0.1

I have made the above changes in lscparams and accepted the changes in SDF.OBSERVE file.

FAMIS 19658

BSC CPS:

ETMY_ST2_CPSINF_V2 high freq noise is high!

HAM CPS:

All look nominal

FAMIS 19976

The anteroom humidity sensor seems to have been revived when we power cycled it during last week's incursion, however after that day the differential pressure between the laser room and the anteroom looks a bit noisier and was not fixed during Jason's incursion yesterday. Perhaps the airflow between the rooms has changed somehow?

J. Kissel, T. Shaffer

TJ noticed that the CAL_AWG_LINES guardian node went into error after Corey and I started the calibration suite this morning (LHO:69684).
After some digging and trending, we realized 
    (1) during yesterday's OMC front-end code restarts, and several DAQ restarts (LHO:69655) -- the CAL_AWG_LINES had obliviously remained in its LINES_ON state. 
    (2) Across several of yesterday evening's post-maintenance lock re-acquisitions, the ISC_LOCK guardian merely re-requested the CAL_AWG_LINES "LINES_ON" state. Since CAL_AWG_LINES was already in the LINES_ON state, no action was taken. However,
    (3) Today, when taking ISC_LOCK to NLN_CAL_MEAS, which requests CAL_AWG_LINES to go to IDLE (through TURN_LINES_OFF), it tried to access the awg test point it had started May 16 2023 04:47:00 UTC (i.e. Monday night, May 15 2023 21:47:00 PDT, likely when the PEM team turned the lines back on after being done for the night), the code fell over.

(A) There's nothing to be really sad about here. It just means we missed a few thermalizations.
(B) I don't think it's urgent to somehow *make* the calls to python / guardian / AWG system robust across computer reboots and DAQ restarts, though see thoughts below
(C) Hopefully, I'll have enough of a plan on what to do about the IFO's thermalization (continuing with all the actions in LHO:69593), that I won't need this guardian
(D) The good enough "solution" is for a human to check the logs, confirm it died because it couldn't access test points that don't exist, and "just" reload the guardian just before we're done with our calibration suite in NLN_CAL_MEAS. I would expect this only needs doing across a Tuesday, and there's only one of those left before the observing run.

In the fullness of time, though, if we do want to start using guardians to drive awg, we will need to figure out a way around this. 
For example, currently, as the CAL_AWG_LINES and awg processes are written, we can't "just" ask the guardian to cycle through its LINES ON and LINES OFF state, because the code is in error. It's also not really good practice for a guardian manager node (in this case ISC_LOCK) to "just assume" what's going on and toggle the LOAD button.

 FRS27187 was opened 9th March 2023 to cover a spate of ADC errors on h1seih16's 2nd ADC. These errors went quiet for a while. We got another error at 00:17 this morning (Wed 17may2023). I have updated the ticket and we will monitor to see if this ADC should be swapped out.

J. Kissel, C. Gray

After Monday's not-insignificant change in PCAL displacement estimate (LHO:69539), to build up the inventory of data with the detector in a stable (and thermalized) configuration, and to give the operator corps more practice on measuring something we'll need to be doing at least once a week during the run, I've asked Corey to take us out of observing to run a 1.5 hour full suite of calibration measurement following the instructions in the OPS wiki.

The time has been cleared with the PEM team.

Time of start is 2023-05-17 15:16 UTC.

TITLE: 05/17 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 126Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 1mph Gusts, 0mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:

Walking in to an Observing H1 with a lock clock of just over 7.5hrs and range hovering between 125-130Mpc.  No alarms.  Little bit of red on the CDS Overview (SEIH16 has a red ADC & CALEY has red from EXC & OVF).  Seismic & wind are quiet.  No Wind Fence Work today!

Switched GRD_IFO from Automatic to Managed.

"PEM Week" continues.

To reconcile SDF differences in the PI damping infrastructure, I've unmonitored the PI PLL Integrators from observe.snap, see the sdf table and the medms, where just the PLL FREQ2 "FM3" integrator filters are now un-monitored in SDF for modes 24, 29, and 31. These integrators are guardian-controlled, so this is probably okay.

TITLE: 05/16 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
SHIFT SUMMARY:

Main Notes:

- PEM injections/SQZ work early on in the day

- Couple locklosses - both due to unknown reasons

- Currently sitting at MAX_POWER on the way back to NLN, I have set the OBSERVING bit to AUTO so it should catch when the SDFs are cleared

- Lock #1:

- LOCKLOSS @ 23:31 UTC - reason currently unknown

- Lock #2:

- Acquired NLN @ 00:52, In OBSERVING @ 4:02

- SQZ/PEM injections ongoing

- When trying to transition to OBSERVE, I ran into some SDF diffs for CALEY - accepting, Tagging CAL 

- LOCKLOSS @ 05:38 - DCPD saturation, reason unknown

- Lock #3:

- Automation failed 3x trying to ACQUIRE_PRMI, would catch for a bit then fail when the ASC engages, intervened next time around and moved PRM ~17 microradians in Y (353.3 --> 336.6) which allowed it to catch

- INCREASE FLASHES on X seems to work good, but sometimes fails to offload when flashes look good enough to catch
LOG:

Catching up on issues since computer restarts, lvea temp drifts, and bringing SHG powers back to normal. IFO was lightly thermalizing its last 5kW in the course of these measurements, taken ~2.5-3 hours after reaching NLN. I think increasing generated SQZ levels from about 9 dB --> 14dB (NLG 4-->9) increased high-frequency squeezing by ~0.3-0.6dB.

• no sqz 1368329184 - 1368329794 (10 min)
  • Took this time to reset our SQZ AS42 no-sqz offsets (had been reset last week in computer restarts), offload all our FC+SQZ ASC outputs to the ZM+FC slider P/Ys, and accepted all this in Observe.snap SDF. Also accepted SQZ-ASC_WFS_SWITCH = OFF in safe.snap SDF of ASCSQZIFO, hope this resolves LHO:69674 so computer restarts don't rail ZMs.
• re-inject sqz. First, I walked the ZM4 sliders from ZM4 P/Y = -900,900 to ZM4 P/Y = -120,-170, with SQZ-ASC running to keep ZM5/6 aligned. Updated slider values saved in SDF, both observe and safe .snaps.
  • These were values I found from when we first opened gate values in Nov. 2022, when maybe the ZM4 --> ZM5 centering was more reasonable? Unsure, we can also try walking ZM4 around, to vary the beam spot over ZM5's surface, and see if anything happens. I had previously tried to offload ZM6 sliders to ZM4, not thinking about the beam spot on ZM5. I had already un-done this big offload 1-2 weeks ago, but last week's computer restarts reset them, so now I'm again bringing ZM4 sliders closer to in-chamber centered values. Recent LLO:64886 saw there was possibly a bad spot to hit ZM5.
• NLG = 3.9, Gen SQZ = 9.2 dB, same OPO TRANS as before, so same NLG uncertainties. Sqz levels below are measured using DTT cursors at 1.4kHz.
  • anti-sqz ~ +6.7 dB at NLG = 3.9, Gen SQZ = 9.2 dB, 1368330725 (angle @ +276).
  • sqz ~ -3.0 dB, 1368328338 (200s, angle @ +161 deg)
• turned up NLG = 9.7, Gen SQZ = 14 dB. Tuned OPO temp to maximize CLF_RF6_DEMOD_RFMON.
  • asqz ~ +11.5 dB. 1368331155 - 1368331177  (angle @ +276)
  • sqz ~ -3.4 dB sqz (angle @ +172.56 deg)
• Re-conciled SDF's so they can proceed to observing. See total trends of this time and the DTT here. More analysis of these sqz/asqz levels to come.

Lockloss a while after. While IFO was down, I re-aligned fiber polarizations given the recent LVEA temp drifts:
   -  opo pump fiber (OPO GRD 'LOCKED_CLF_DUAL_NO_ISS', minimize in-vac rejected power H1:SQZ-SHG_FIBR_REJECTED_DC_POWER)
   -  clf fiber (CLF GRD 'DOWN', maximize CLF_REFL_LF_OUT_DQ)
   -  fcgs fiber (doesn't matter GRD state, I minimized in-vac rejected power H1:SQZ-FC_FIBR_REJECTED_DC_POWER. This one was pretty close.)

Maybe sometime we can check the SQZ ASC gain / time constant and make sure it makes sense, since its outputs seem to move a lot very quickly? Also, since IFO was still thermalizing, if we have no-sqz time with a very thermalized IFO, it might be worth re-setting the AS42 no-sqz offsets again then.

Ryan C., Ibrahim, Austin

This alog is a follow up to a previous log showing that in every acquisition of NLN, particularly about 7/8 mintues in, the ADS will suddenly drift and signals (most prominent in ADS YAW 3) will start to diverge, and prevent us from going into OBSERVING for ~15 minutes per lock. The magnitude of this drift varies depending on each lock acqusition, but this has been a consistent issue for a while now. With the help of Ryan C and Ibrahim, here are our updates findings/what we looked into.

We decided to watch a "live" version of this drift when we acquired NLN and decided to trend a myriad of channels of anything we thought could be worth looking into namely: all oplevs (with the exception of SR3) - screenshot attached - (sorry for x axis looking terrible I couldn't get the scale right), ASC CAMERA SERVO/ERROR signals - screenshot attached, PRM/IM4 witness sensors (based off our findings from the previous log, we were curious how much these suspensions were moving, spoiler - not a whole lot), LSC PRC1, and ASC PRC 2. When reviewing all of these channels when an ADS kick occurred, most checked out ok, but I have attached the CAMERA SERVO error signals as that did move a bit, but not by much.

We then looked directly in the PRM suspension and tried to find any clues, and it turns out we found something interesting. We found that the PRM DRIVE ALIGN M1/M3 stages behave in the exact same fashion as the ADS drift (ss 1 / 2). Is this expected? Might be, since we know that ADS YAW 3 feeds directly into PRM. But we thought it was worth noting here.

In addition, we found something regarding the CAMERA_SERVO guardian. When the CAMERA_SERVO guardian is in state 400 - TURN_CAMERA_SERVO_ON, the issue seems to be most apparent (screenshot). This coincides with the CAMERA guardian and to us makes sense since in that state, many of the ADS settings are changed. However, there does sometimes appear to be some movement (a "dip") before the CAMERA_SERVO guardian actually gets to this state (which can be seen here). If this "dip" feature is causing the drift that we've been seeing it might not be the CAMERA_SERVO guardian since these dips happens when the guardian is in DOWN and not doing any actuation. But, it could be entirely possible that this dip feature is just a red herring too and is a biproduct of the ADS converging. But maybe worth looking into.

So when did this problem start? After trending back a couple weeks and looking at every NLN acquisition, we found that this issue started around April 28th. From the image, you can see that on the lock acquisitions before the 28th, things were mostly quiet and there were no kicks/dips on the drive align signal. But afterwards, it becomes apparent that something has changed. After doing some alog digging around this time, I saw that there was a change to the ADS gains - alog here, acutal changes can be seen here (the change was made on 4/28, but was not committed to svn until 5/6). Though, I'm not entirely convinced this is the culprit, but it might be worth to see what the ADS signals would look like at a NLN lock if we reverted these gain values.

With all this in mind, we plan to take some next steps into narrowing down the issue, by looking more into the CAMERA_SERVO guardian code, figuring out why the kick happens specifically 8 minutes into a lock, and finding other correlated signals that we might have missed (though after seeing what we saw for the better part of the past week, I'm leaning towards the issue being related to the CAMERA guardian). Stay tuned!

Extreme 1-month late post, but probably worth logging even if late. A month ago on April 16, 2023, I turned down the power on OPO REFL's RF80 PD, to not saturate the refl diode we use for opo locking. I noticed that OPO_REFL_DC_VOLTS was saturated at 10V, (like OPO_REFL_DC_POWER ~ 4.3mW), so I pico-waveplate'd to dump power away from the RF PD, towards OPO_REFL_DC_REJECTED_POWER. At first, the OPO_REFL_RF80_DEMOD_RFMON went up as the optical power decreased (bad/weird), then finally the RFMON went down as the optical power decreased, which I think is expected if we're not saturating the diode.

The IFO has been locked for ~2 hours and PEM injections are currently ongoing.

After some restarts this morning, SQZ-ASC came back online and caused SUSSQZOUT saturations from ZM5/6. See trends. It looks like the SQZ ASC switch restarted as "on", and an input signal got sent to the SQZ ASC filter banks (top trend). Our ASC filter banks integrated up this signal (2nd trend), and pushed on ZM5/6 (last trend), leading to verbal saturation alarms.

What I did: Go to SITEMAP > SQZ > SQZ ASC IFO, turn asc to OFF, and then click the "graceful clear history" purple button on top to clear the filter bank outputs. You can also go to SITEMAP > SQZ > SQZ OVERVIEW > "IFO ASC" black button in the middle >> click "OFF". Then click the IFO ASC black button to open the sqz asc medm screen to "graceful clear history".

Problem & solution: The H1:SQZ-ASC_WFS_SWITCH was not monitored in safe.snap SDF, and so it came back online as ON. I have monitored this asc switch now in safe.snap, and will accept it as "OFF" in the next opportunity. Weirdly, in Observe.snap, this asc switch is properly monitored and accepted as "ON", so everything is fine there.

TITLE: 05/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:

Started Maintenance Recovery at about 1:53pmPT with (3) issues which made recovery tough, but help from Jeff & TJ we were able to get to NLN 342pmPT.

LOG:

• 1445 Magnetic Injection
• 1446 PI 24
• 1458 Forklifting around corner (tyler & christina)
• 1458 Wind Fence work (mitch)
  • Transitioned EY to "Wind Fence" mode
  • After lockloss took SEI_ENV to MAINTENANCE
• 1459 Lockloss
• 1508 Turbo pump famis at EX/MX/MY (jordan)
  • 1827-1852 Powering turbos off (jordan)
• 1510 Took ISC LOCK to IDLE & IMC_LOCK to OFFLINE
• 1519-1742 Turbo pump famis at LVEA (gerardo.janos)
• 1512 OBSERVATORY MODE to Preventative Maintenance
• 1523 Power Supply swap at RF rack prep:  ASC0, LSC0, OMC0 models stopped AND IO Chassis powered off (dave, jeff)
  • PSL OMC/FSS affected by RF
  • ISI CPS' watchdog trips in Corner due to timing
• 1525-1807 EY PCal (tony, jenny)
  • 1558 Laser HAZARD AND ALS laser was shuttered
  • 1807 Laser SAFE and ALS shutter OPENED.
• 1524-1615 EY cleaning (cindy, kim)
  • 1641-1849 lvea cleaning 
• 1528-1858 EY hepta pump (travis)
• 1530 Magnetometer & cable pulling around HAM7/sqz stuff (fil)
  • 1542 richard meeting fil
• 1532-1712 fss alignment at PSL (jason)
  • For restoring (after RF work), jason mentioned that autolockers didn't work for him
• 1551 Beckhoff work to "make screens green" (daniel)
• 1557-1847 FAMIS checks (soike)
• 1556 Access system work in lvea/Ham Shack (APS, richard)
  • 1615 2nd truck arrives
• 1608 HAM2/3/7/8 SEI meas (jim)
• 1600 First successful SNEWS alert test
• 1624 Beckhoff reboot (daniel)
  • 1702, 1704 more reboots
• 1640 EX fan check/changes (bubba)
  • 1710 Didn't notice anything on inspection, but will switch fans via FMCS
• 1643 Hanford fire inspection up/down arms; no bldg incursions (tyler)
• 1652 Lawn service contractor
• 1657-1715 Cosmic Ray set-up/checks (adrian)
• 1710 Noticed UNIFIRST card swipe in on camera
• 1719 Power Supply work for ISC C1 & C2 racks (marc.fernando)
• 1724 All corner ISI Watchdogs tripped due to CPS (due to power supply work?)
  • 1727 Jim going out to check---
• 1730 Dust Monitor #6 alarms
• 1742 EX getting accels working and packing up injection stuff (robert.anamaria)
• 1742 PSL (#102) dust alarm.
• 1809 EY Hartman laser powered on for Camilla (tony)
• 1816-1832 EX Hartman laser power on (Camilla)
• 1900 
• 1915 Beginning to restore from power supplies & DAQ restart (dave).
• 1917 HVAC at EX being shut down (robert)
• 1953 Beginning to Restore H1 after Tues Maintenance!
• 1953-1955 Dark Offset Script---reason for this was the Power Supply/RF rack/Slow Controls work on the ISC Racks during Maintenance (alog 69652) 
  • IMC Not Locking:  IMC could not lock after this (MC Trans was noticeably off in yaw); This was due to the Power Supply/Front End/rack work.  To get this back, we adjusted the PSL PZTs while watching the MC2 Trans & IM4 Trans channels.
    • REASON FOR ISSUE:  Powering off racks during Power Supply swap caused PZT misalignment
  • Y-Arm Green Alignment Issue:  With an UNLOCKED Y-Arm, Flashes just couldn't get above a certain level via ETMy & TMSy.  Took the NON-STANDARD step of touching the ITMy--Because all ISIs had an ISI CPS watchdog trip due to the ISI's using Timing from the ISC.  ITMy yaw slider was moved to -13.7 (from -12.8).
    • REASON FOR ISSUE: ISI Watchdog Trip causing small misalignmen t for ITMy
  • Initial Alignment X-Arm Issue during Input Align:  For Initial Alignment's Input Align step, could not lock the X-arm.  It was noticed the +21 Whitening gain [H1:LSC-POP_A_RF45_WHITEN_GAIN] was the issue with the X-Arm locking---This was because of the Dark Offset script running earlier (that was done with the standard +15 Whitening gain---which is used for everything EXCEPT Initial Alignment!).  And so H1:LSC-POP_A_RF45_I_OFFSET was changed from -32.677 to -70 while zeroing H1:LSC-POP_A_RF45_I_ERR_DQ while watching it on ndscope. 
    • REASON FOR ISSUE: Dark offsets & Whitening gain being different for Initial Alignment vs all other states! 
  • Separate NOTE:  For next LockLoss, want to see if the IMC remains locked (due to guardian change made by TJ).
• 2132-2310 Squeezer local hazard (camilla, naoki, vicky)
• 2242 NOMINAL LOW NOISE, but no Squeezing.

Jennie, Sheila

Sheila and I looked at the steps Elenna changed the DARM offset through, the last time the contrast defect measurement was taken (see LHO #69361). To get an idea of how much light seen at the anti-symmetric port (calibrated in terms of power into HAM6) is insensitive to DARM motion I plotted the scaling of the light at ASC-AS_C_NSUM_OUTPUT with DARM offset changing (OMC-DCPD_SUM_OUTPUT).

Attached is the plot (pdf) of total power into HAM 6 versus the power that gets through the OMC.

The uncertainty in the AS measurement is was estimated using the y cursors on ndscope (shown in png).

0.837W of power is predicted for no DARM offset, and 82% of the light coming into HAM 6 is sensed by the OMC DCPDs.

We also tried to use the total power on OMC REFL to do a similar calculation of the light rejected from the OMC that is insensitive to DARM motion but due the noise on this signal we made need bigger DARM offset steps to see a clear trend.

Since this measurement, whitening has been implemented on the OMC REFL PD by Daniel.

Code is Plot_OMC_REFL_2023-05-12.ipynb in /ligo/home/jennifer.wright/git/OMC_mode_matching/

Figure is in /ligo/home/jennifer.wright/git/OMC_mode_matching/figures

Gabriele, Camilla, Elenna

We have not updated the in-loop LSC feedforward since the increase in the DARM offset and the new SRCL offset (also small increase in MICH gain). These changes had a large effect on the feedforward, and Jenne has been able to subtract some but not all of the LSC noise using nonsens cleaning. I made three injections last night that would allow Gabriele to fit the feedforward and update the filters. I ran a MICH and SRCL injection, as well as a filter bank injection (input off, filters off, gain 1) to get the other preshaping effects in the feedforward. I did not run a PRCL injection, as we will need to update MICH and SRCL first, and then measure PRCL. The PRCL fit will depend on the MICH and SRCL feedforward.

The injection measurements are saved in "/ligo/home/elenna.capote/LSCFF".

Both new filters are labeled "5-16-23" and are placed in FM9 of MICHFF and SRCL1FF respectively. These filters should be used with a gain of 1. I updated the ISC_LOCK guardian to use these filters in lownoise_length_control.

Given that the PRCL feedforward is so dependent on MICH and SRCL FF, in the event that we are unable to implement a new PRCL FF in a timely manner, it is worth checking if the current PRCL feedforward is still improving noise or injecting noise with the MICH and SRCL update.

While LLO were not Observing, I took some SQZ data to help better understand our SQZ angle for future overnight tests. Our squeezing hasn't been great over the last few days, we expect as our NLG and power though the SHG have decreased, we will go onto SQZT0 tomorrow.

• 6:19 to 6:30 UTC, I made small SQZ angle adjustments during Observing tagging DetChar. See attached plot and dtt.
• 6:33 to 6:43 NoSQZ. DTT plot
• 6:45 to 6:52 FIS (SQZ_MANAGER went to FIS via FDS, I think this may be because I weighted the edges between SQZ_ASC and FIS, we should maybe remove this)
• 6:54 Back to FDS (Took two attempts of FC locking to go back to FDS.)
• 6:54 - 7:16 Finding SQZ (161deg), No-SQZ (positive CLF servo sign 97deg / normal sign 272 was nearly as much A-SQZ) and mid-SQZ (225deg) levels, attached plot. 9dB difference between SQZ and Anti-SQZ.