TJ, Dave:

We have installed a new EPICS IOC which hosts PVs to support OPS stand-downs due to astrophysical events reported by external agencies (e.g. GraceDb).

The IOC runs on cdsioc0, is under systemd process control and is configured by puppet.

I have created a simple MEDM (see attachment) for testing. In the example shown it is reporting the system in standdown mode. The IOC reports the event's details, along with a calculated time-to-end.

TITLE: 05/17 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 9mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.11 μm/s 
QUICK SUMMARY:

- IFO is relocking (currently @ DHARD_WFS)

- Once back up to NLN, we plan to continue with PEM injections

- CDS/SEI/DMs ok

TITLE: 05/17 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 120Mpc
SHIFT SUMMARY:

Fairly basic shift with one lockloss which came back on an acquisition without any intervention (it took 64min.)  Started day off with Calibration time and then PEM took over after H1 was relocked.  Had one lockloss in last hour which took a few attempts to get past DRMI (& handed off to Austin).
LOG:

• 1459 EY clean-up, no large equipment movement--so no "Fence Work" for EY ISI (randy)
• 1511 Optics Lab cleaning (kim)
• 1516-1700 Calibration suite (corey)
• 1615-1700 Pcal lab (tony)
  • Multiple incursions after
• 1631-1720 PEM Prep:  Robert to EX to pick up items, pick up Anamaria and then off to EY
  • 1653 Heading to EY
• 1707 Lockloss (probably PEM since we see a step up for EY  on 0.03-0.1 band around same time)
• 1731-1757 SQzt0 with local hazard (camilla, vicky)
• 1707 LOCK #1
  • No Intervention until PRMI acquisition....it looked dead, so after a few minutes, requested CHECK_MICH_FRINGES.  When this was done, requested PRMI_ASC.
  • 1751 LOCKLOSS during ACQUIRE_PRMI
• 1751 LOCK #2
  • Green Y needed INCREASE_FLASHES
  • 1854 NLN & PEM take H1.
• 1756-1835 Checking hepta headers in lvea (gerardo)
• 1800-1806:  Optics lab visit (jason)
• 1925-1940:  Wayward truck arrived and turned away (backed up and exit via path in front of LSB & LExC).  Also contacted Bubba & he talked with driver on his way out.
• 1955-2017:  Randy moving small parts and taking inventory.
• 2000 Mary Jane returns to the site for filmmaking work.
• 2009 Observed "picket fence" stopped (came back after TJ logged into the nuc).
• 2019 Change to HAM3 HEPI (jim)
• nuc30's DARM dtt stopped.  Clicking START, crashed the window.
• 2207 LOCKLOSS
• 2207 LOCK #3:
  • Lockloss at FINDIR
• 2217 LOCK#4
  • IR Not Found for ALS_DIFF, so walked DIFF OFFSET to get flashes on Y-arm
  • lockloss at acquire prmi
• 2232 Lock #5
  • lockloss at find ir
• 2237 lock#6
  • lockloss at prmi
• 2246 lock#7
  • Took directly to PRMI, and as I was getting ready to move PRM, PRMI locked...just as Austin was walking in the Control Room....and so I'm handing off to him.

Vicky, Camilla

We adjusted the alignment into the OPO pump fiber on SQZT0 but touching BG2 and MG3 (layout on D1201210) to increase power coupling through fiber by ~5%, see attached. Vicky noticed OPO REFL was lower than usual, with 14mW into the OPO pump fiber, we were getting 1.5mW on SQZT7 OPO REFL and 1.0mW REFL Rejected and 50% of total goes to SK path (BS in HAM7), so total of 5mW/14mW = 35% throughout. Nominal Launch power is ~20mW.

Launch power was reducing due to temperature change of having table enclosure open. We pico-ed the HWP (pico I motor 2) to reduce the power into the fiber while we did this.

After the calibration update this morning, I need to retrain the NonSENS noise cleaning.  However, the NDS team is still in progress on understanding why we can't get past GDS-CALIB_STRAIN channels right now.  Once we can get that past data, I should be able to retrain.  Until then, the NOISE_EST is zeros, so GDS-CALIB_STRAIN_CLEAN will be the same as GDS-CALIB_STRAIN_NOLINES.

The h1sqz and h1sqzfces SDFs have identical safe and observe snap files (and always had).

h1ascsqzfc and h1ascsqzifo did have separate files which were more or less idential. I changed the link of the observe files to point to the safe ones.

All squeezer related SDFs are now using identical observe and safe files.

J. Kissel, L. Dartez, J. Rollins
All dates called out "verbally" rather than automatically generated are in the format YYYY-MM-DD. 
All dates embedded into file names, automatically generated, are YYYYMMDD(T)HHMMSS(Z) where "T" represents "Time" and "Z" represents "Zulu" aka UTC time.

Setting aside our desire to have immediate feedback that "what we've done is good" using a thermalized IFO that's not otherwise in use, we've updated the calibration pipeline. 

This physics of this update includes:
    - Minor changes to the CAL-CS foton filters that represent the calibration model's "free parameters." These are needed because the last update to the sensing function parameters was on May 05 2023 (LHO:69332), using sensing function data from 2023-05-04 -- prior to us finalizing the -290 SRCL offset on May 08 2023 LHO:69402, and the DARM offset equivalent to 40 mA of OMC DCPD SUM LHO:69358. Note that these updates do NOT include measurements taken after, or corrected for the 0.6% systematic error that had been present in PCAL and only corrected for on Friday May 12 2023 (LHO:69539)
        :: a new (inverse) optical gain, and cavity pole (actually different because of the new SRCL and DARM offsets)
        :: a new set of actuator strengths for the L1, L2, and L3 stages (only different because there's noise in the MCMC fitting)
    - Minor changes to "modeled DARM loop transfer functions at calibration line frequencies" EPICs records, corresponding to the above changes in the free parameters
    - Updates to final stages of FIR filters within the GDS (DMT) portion of the calibration line related to noise cleaning; no changes were made to the "primary" FIR filters which correct the sensing and actuation paths (since these do not depend on the free parameters).

The new report also comes with, for the first time with the modern pydarm cmd-dev infrastructure, trust-worthy estimates of our "unknown frequency systematic error" [GPR fit results of the sensing and actuation functions]. That should mean we have everything we need to produce a modeled estimate of the response function systematic error (with uncertainty, sometime just referred to as "the uncertainty budget") though, only for frequencies below 1000 Hz, and for times when the IFO is completely thermalized.

#LibrarianshipDetails

This update uses the values and filters from the 20230510T062635Z calibration report (attached, in case it gets overwritten in the future), derived from the collection of "fixed" pydarm model parameter values set by the pydarm_H1.ini parameter file whose version has git hash c0bd4afe, and MCMC fits to the free parameters from 
    - sensing measurements taken on 2023-05-10, LHO:69462
        MCMC chains saved to 
            ~cal/archive/H1/reports/20230510T062635Z/
                sensing_mcmc_chain.hdf5
    - actuation measurements taken on 2023-05-09, LHO:69427
        MCMC chains saved to 
            ~cal/archive/H1/reports/20230510T062635Z/
                actuation_L1_EX_mcmc_chain.hdf5
                actuation_L2_EX_mcmc_chain.hdf5
                actuation_L3_EX_mcmc_chain.hdf5
and generated with the cmd-dev branch of the pydarm software suite, which has git hash 048acc12d.

The GPR fits also live report's folder, 
    ~cal/archive/H1/reports/20230510T062635Z/
        sensing_gpr.hdf5

        actuation_L1_EX_gpr.hdf5
        actuation_L2_EX_gpr.hdf5
        actuation_L3_EX_gpr.hdf5

Took H1 out of Observing to run Calibration Suite Measurements.

• Took us out of OBSERVING, Observatory Mode to CALIBRATION, & took ISC_LOCK to NLN_CAL_MEAS
• Opened terminal and entered/ran:  "pydarm measure --run-headless" at 1516utc.
• Measurement completed at 1700utc.

Attached is a screenshot (#1) of the sitemap > CAL CS > Calibration Monitor screen shortly after starting the measurement.

• Measurement at:  /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20230517T151727Z.xml

'pydarm report' command gave attached report: 

• /ligo/groups/cal/H1/reports/20230517T163625Z/H1_calibration_report_20230517T163625Z.pdf (attachment #2)

Since, PEM team was going to immediately take H1, did the following:

• CAL_AWG_LINES "loaded" to clear RED Error state (jeff)
• ISC_LOCK:  NLN  (which turns CAL_AWG_LINES back on)
• CAL_AWG_LINES turned OFF by going to IDLE (jeff)
• Observatory Mode switched to COMMISSIONING
• H1 ready for PEM team

NOTE:  CAL_AWG_LINES Goes Into Error (see attached screenshot [#3])

While getting this measurement going, the guardian CAL_AWG_LINES went into ERROR (node is RED).

(I missed this while setting up measurement and then starting draft of this alog.  TJ noticed it when he walked in.  I should have noticed it from Verbal or from the notification messages on various Guardian windows.)

Jeff mentions that this can happen after there has been a DAQ Restart (there were atleast two yesterday). To get the node back, Jeff hit the LOAD button on CAL_AWG_LINES.

Wed May 17 10:08:44 2023 INFO: Fill completed in 8min 44secs

Jordan confirmed a good fill curbside.

I have found a new setting for ITMY mode 07 since the nominal settings was making it grow (past couple of locks). Given below are the settings which works - see ndscope attached  (old settings shows the mode is growing and the new one drops quickly) below,

New: FM1+FM10, Gain 0.1

Old: FM1+FM2+FM10, Gain 0.1

I have made the above changes in lscparams and accepted the changes in SDF.OBSERVE file.

FAMIS 19658

BSC CPS:

ETMY_ST2_CPSINF_V2 high freq noise is high!

HAM CPS:

All look nominal

FAMIS 19976

The anteroom humidity sensor seems to have been revived when we power cycled it during last week's incursion, however after that day the differential pressure between the laser room and the anteroom looks a bit noisier and was not fixed during Jason's incursion yesterday. Perhaps the airflow between the rooms has changed somehow?

J. Kissel, T. Shaffer

TJ noticed that the CAL_AWG_LINES guardian node went into error after Corey and I started the calibration suite this morning (LHO:69684).
After some digging and trending, we realized 
    (1) during yesterday's OMC front-end code restarts, and several DAQ restarts (LHO:69655) -- the CAL_AWG_LINES had obliviously remained in its LINES_ON state. 
    (2) Across several of yesterday evening's post-maintenance lock re-acquisitions, the ISC_LOCK guardian merely re-requested the CAL_AWG_LINES "LINES_ON" state. Since CAL_AWG_LINES was already in the LINES_ON state, no action was taken. However,
    (3) Today, when taking ISC_LOCK to NLN_CAL_MEAS, which requests CAL_AWG_LINES to go to IDLE (through TURN_LINES_OFF), it tried to access the awg test point it had started May 16 2023 04:47:00 UTC (i.e. Monday night, May 15 2023 21:47:00 PDT, likely when the PEM team turned the lines back on after being done for the night), the code fell over.

(A) There's nothing to be really sad about here. It just means we missed a few thermalizations.
(B) I don't think it's urgent to somehow *make* the calls to python / guardian / AWG system robust across computer reboots and DAQ restarts, though see thoughts below
(C) Hopefully, I'll have enough of a plan on what to do about the IFO's thermalization (continuing with all the actions in LHO:69593), that I won't need this guardian
(D) The good enough "solution" is for a human to check the logs, confirm it died because it couldn't access test points that don't exist, and "just" reload the guardian just before we're done with our calibration suite in NLN_CAL_MEAS. I would expect this only needs doing across a Tuesday, and there's only one of those left before the observing run.

In the fullness of time, though, if we do want to start using guardians to drive awg, we will need to figure out a way around this. 
For example, currently, as the CAL_AWG_LINES and awg processes are written, we can't "just" ask the guardian to cycle through its LINES ON and LINES OFF state, because the code is in error. It's also not really good practice for a guardian manager node (in this case ISC_LOCK) to "just assume" what's going on and toggle the LOAD button.

J. Kissel, C. Gray

After Monday's not-insignificant change in PCAL displacement estimate (LHO:69539), to build up the inventory of data with the detector in a stable (and thermalized) configuration, and to give the operator corps more practice on measuring something we'll need to be doing at least once a week during the run, I've asked Corey to take us out of observing to run a 1.5 hour full suite of calibration measurement following the instructions in the OPS wiki.

The time has been cleared with the PEM team.

Time of start is 2023-05-17 15:16 UTC.

TITLE: 05/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:

Started Maintenance Recovery at about 1:53pmPT with (3) issues which made recovery tough, but help from Jeff & TJ we were able to get to NLN 342pmPT.

LOG:

• 1445 Magnetic Injection
• 1446 PI 24
• 1458 Forklifting around corner (tyler & christina)
• 1458 Wind Fence work (mitch)
  • Transitioned EY to "Wind Fence" mode
  • After lockloss took SEI_ENV to MAINTENANCE
• 1459 Lockloss
• 1508 Turbo pump famis at EX/MX/MY (jordan)
  • 1827-1852 Powering turbos off (jordan)
• 1510 Took ISC LOCK to IDLE & IMC_LOCK to OFFLINE
• 1519-1742 Turbo pump famis at LVEA (gerardo.janos)
• 1512 OBSERVATORY MODE to Preventative Maintenance
• 1523 Power Supply swap at RF rack prep:  ASC0, LSC0, OMC0 models stopped AND IO Chassis powered off (dave, jeff)
  • PSL OMC/FSS affected by RF
  • ISI CPS' watchdog trips in Corner due to timing
• 1525-1807 EY PCal (tony, jenny)
  • 1558 Laser HAZARD AND ALS laser was shuttered
  • 1807 Laser SAFE and ALS shutter OPENED.
• 1524-1615 EY cleaning (cindy, kim)
  • 1641-1849 lvea cleaning 
• 1528-1858 EY hepta pump (travis)
• 1530 Magnetometer & cable pulling around HAM7/sqz stuff (fil)
  • 1542 richard meeting fil
• 1532-1712 fss alignment at PSL (jason)
  • For restoring (after RF work), jason mentioned that autolockers didn't work for him
• 1551 Beckhoff work to "make screens green" (daniel)
• 1557-1847 FAMIS checks (soike)
• 1556 Access system work in lvea/Ham Shack (APS, richard)
  • 1615 2nd truck arrives
• 1608 HAM2/3/7/8 SEI meas (jim)
• 1600 First successful SNEWS alert test
• 1624 Beckhoff reboot (daniel)
  • 1702, 1704 more reboots
• 1640 EX fan check/changes (bubba)
  • 1710 Didn't notice anything on inspection, but will switch fans via FMCS
• 1643 Hanford fire inspection up/down arms; no bldg incursions (tyler)
• 1652 Lawn service contractor
• 1657-1715 Cosmic Ray set-up/checks (adrian)
• 1710 Noticed UNIFIRST card swipe in on camera
• 1719 Power Supply work for ISC C1 & C2 racks (marc.fernando)
• 1724 All corner ISI Watchdogs tripped due to CPS (due to power supply work?)
  • 1727 Jim going out to check---
• 1730 Dust Monitor #6 alarms
• 1742 EX getting accels working and packing up injection stuff (robert.anamaria)
• 1742 PSL (#102) dust alarm.
• 1809 EY Hartman laser powered on for Camilla (tony)
• 1816-1832 EX Hartman laser power on (Camilla)
• 1900 
• 1915 Beginning to restore from power supplies & DAQ restart (dave).
• 1917 HVAC at EX being shut down (robert)
• 1953 Beginning to Restore H1 after Tues Maintenance!
• 1953-1955 Dark Offset Script---reason for this was the Power Supply/RF rack/Slow Controls work on the ISC Racks during Maintenance (alog 69652) 
  • IMC Not Locking:  IMC could not lock after this (MC Trans was noticeably off in yaw); This was due to the Power Supply/Front End/rack work.  To get this back, we adjusted the PSL PZTs while watching the MC2 Trans & IM4 Trans channels.
    • REASON FOR ISSUE:  Powering off racks during Power Supply swap caused PZT misalignment
  • Y-Arm Green Alignment Issue:  With an UNLOCKED Y-Arm, Flashes just couldn't get above a certain level via ETMy & TMSy.  Took the NON-STANDARD step of touching the ITMy--Because all ISIs had an ISI CPS watchdog trip due to the ISI's using Timing from the ISC.  ITMy yaw slider was moved to -13.7 (from -12.8).
    • REASON FOR ISSUE: ISI Watchdog Trip causing small misalignmen t for ITMy
  • Initial Alignment X-Arm Issue during Input Align:  For Initial Alignment's Input Align step, could not lock the X-arm.  It was noticed the +21 Whitening gain [H1:LSC-POP_A_RF45_WHITEN_GAIN] was the issue with the X-Arm locking---This was because of the Dark Offset script running earlier (that was done with the standard +15 Whitening gain---which is used for everything EXCEPT Initial Alignment!).  And so H1:LSC-POP_A_RF45_I_OFFSET was changed from -32.677 to -70 while zeroing H1:LSC-POP_A_RF45_I_ERR_DQ while watching it on ndscope. 
    • REASON FOR ISSUE: Dark offsets & Whitening gain being different for Initial Alignment vs all other states! 
  • Separate NOTE:  For next LockLoss, want to see if the IMC remains locked (due to guardian change made by TJ).
• 2132-2310 Squeezer local hazard (camilla, naoki, vicky)
• 2242 NOMINAL LOW NOISE, but no Squeezing.

Jennie, Sheila

Sheila and I looked at the steps Elenna changed the DARM offset through, the last time the contrast defect measurement was taken (see LHO #69361). To get an idea of how much light seen at the anti-symmetric port (calibrated in terms of power into HAM6) is insensitive to DARM motion I plotted the scaling of the light at ASC-AS_C_NSUM_OUTPUT with DARM offset changing (OMC-DCPD_SUM_OUTPUT).

Attached is the plot (pdf) of total power into HAM 6 versus the power that gets through the OMC.

The uncertainty in the AS measurement is was estimated using the y cursors on ndscope (shown in png).

0.837W of power is predicted for no DARM offset, and 82% of the light coming into HAM 6 is sensed by the OMC DCPDs.

We also tried to use the total power on OMC REFL to do a similar calculation of the light rejected from the OMC that is insensitive to DARM motion but due the noise on this signal we made need bigger DARM offset steps to see a clear trend.

Since this measurement, whitening has been implemented on the OMC REFL PD by Daniel.

Code is Plot_OMC_REFL_2023-05-12.ipynb in /ligo/home/jennifer.wright/git/OMC_mode_matching/

Figure is in /ligo/home/jennifer.wright/git/OMC_mode_matching/figures

Due to some concerns raised by Evan Hall regarding vibration and noise from the # 1 supply fan at End X, I have switched from Fan 1 to Fan 2.