Wed Apr 03 10:07:57 2024 INFO: Fill completed in 7min 53secs

Jordan confirmed a good fill curbside.

Naoki, Eric, Camilla

Before PSAMS scan, we checked the NLG. We followed the instruction in 76542. The OPO temperature seems already optimal. The measured NLG is 0.169/0.00994 = 17, which is consistent with previous measurement in 76540.

cds-crtools was updated to version 4.11 in the CDS conda environment.

The major changes are:

• A bug fix in awggui that could cause a glitch in an exciation when loading a file.
• foton no longer warns by default when a filter has a zero in the right-hand plane.  The warning can be turned on with a command line argument '-Z t', or an environment variable FOTON_WARN_RHP_ZEROS=true.
• The criterion for detecting a zero in the right-hand plane has been relaxed. Zeros that are ostensibly on the imaginary axis, but due to precision issues are slightly in the right-hand plane will no longer be flagged.

A complete change history for all crtools can be found here:

https://git.ligo.org/cds/software/dtt/-/wikis/ChangeHistory

WP11743 SUS DACKILL removal and WD consolidation

Jeff, Oli, Dave

New models were install for h1susim, h1sushtts, h1sussqzin, h1sussqzout, h1susifoout. No SEI model changes were needed, IOP models were not restarted so no SWWD bypass was necessary. DAQ restart was required.

Beckhoff OPLEV upgrade

Fernando, Daniel, Dave:

New OPLEV code was installed on the Beckhoff slow controls, resulting in additional 20 slow channels to H1EPICS_ECATAUX[CS,EX,EY]. EDC+DAQ restart was required.

PSL Neolase Beckhoff code change

Patrick, Ryan S, Dave:

The PSL Diode room Beckhoff code was modified, resulting in one additional channel (H1:PSL-LASER_PDWD) in H1EPICS_PSLNEOLASE.ini. EDC+DAQ restart was required.

DAQ Restarts

Jeff, Dave:

The DAQ was restarted twice. The first around 09:30 immediately after the SUS model changes, and then a second time with an EDC restart around 11:30 to support the Beckhoff changes.

GDS Broadcast List Change:

The h1susifoout change resulted in the removal of 6 fast channels (256Hz) from the frame, of which 2 were being sent DMT via the GDS broadcaster. ECR2400116 was created and approved to cover this removal.

--: fast channel H1:SUS-OMC_M1_ASC_L_IN1_DQ removed from the DAQ 
-!: fast channel H1:SUS-OMC_M1_ASC_P_IN1_DQ removed from the DAQ  ***GDS-CHAN***
--: fast channel H1:SUS-OMC_M1_ASC_R_IN1_DQ removed from the DAQ 
--: fast channel H1:SUS-OMC_M1_ASC_T_IN1_DQ removed from the DAQ 
--: fast channel H1:SUS-OMC_M1_ASC_V_IN1_DQ removed from the DAQ 
-!: fast channel H1:SUS-OMC_M1_ASC_Y_IN1_DQ removed from the DAQ  ***GDS-CHAN***
 

To expediate the broadcast list change, I hand editied the gds_broadcast.ini file on h1daqgds[0,1]. This needs to be made permanent in puppet.

Three fast channels were added to the frame:

++: fast channel H1:SUS-OMC_M1_LOCK_L_IN1_DQ added to the DAQ
++: fast channel H1:SUS-OMC_M1_LOCK_P_IN1_DQ added to the DAQ
++: fast channel H1:SUS-OMC_M1_LOCK_Y_IN1_DQ added to the DAQ
 

Yesterday 4-2-2024 the HVAC pre-filters at each of the 4 out station buildings along the arms were replaced with new pre-filters. The new pre filters are the same size (24"x24"x2") and filtration level (MERV-13) as the old pre-filters that were taken out. 
During inspection of the pre-filters an inspection was done on the "Main Filter Cartridge" which is the harder, more permanent, deeper, and heavier duty material type cartridge. It's installed in front of the pre-filter that is used for air filtration in the HVAC system. These are nearing the end of their life cycle. These will likely be replaced in the coming couple months.

Alerted around 645 PT about initial alignment not completing. I found it with MICH not locking, but flashes seemed a bit off. I tried realigning via the camera and trying to maximize the flashes in AS_A. Even after getting a good spot on camera and having good flashes, it would diverge when tring to lock. MICH dark was the same. I tried to skip this state but SR2 alignment wouldn't even lock, and trying it by hand seemed that there was something very off.

Richard had told me that he set a node to INIT around 630PT, trending back it looked like in Input_Align. We've been having issues with this state recently, I guess that makes sense. I then tried this state and it locked without issue. I then tried PRC, no issues. Back to MICH, same issues as before. Tried skipping it with a well aligned spot on camera, and ran into the same issues with SR2 alignment.

I'm handing off to Corey with the suggestion to restore alignment to the last alignment Oli did and then trying an alignment since I was able to lock with this earlier.

TITLE: 04/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 9mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.23 μm/s
QUICK SUMMARY:

H1 lost lock about 3.5hrs ago (1238utc), and TJ (owl shift-er) has been trying to help H1 with an alignment, BUT the Initial Alignment woes continue.  Strangely, Input Align appeared to work for him, but every other (atleast MichBright and SRC) were not completing for him. 

Will try an overall alignment revert to the alignment Oli/TJ had last night.  And from that will run an alignment to see if Initial Alignment steps still have issues.  (if they do, I'll skip the alignment and try locking with the reverted alignment.

Enviornmentally all is quiet after the windy and shaky night.

Oli handed off the IFO with the wind keeping ALS from locking, but they made it through an initial alignment earlier. Once the wind calmed down a bit ~30min later, it was able to keep ALS locked and then went right up without intervention. 

TITLE: 04/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Relocking from when I showed up went quickly, but after the earthquake knocked us out it's been impossible to lock due to the ground motion and the rising high winds :(
LOG:

23:30 Reached NOMINAL_LOW_NOISE    
23:41 Observing

00:11 Lockloss due to large earthquake from Taiwan (76897), holding in IDLE

Relocking
04:45 Untripped all watchdogs, attempting to lock green arms before starting an inital alignment
    - Green arms are super wiggly still from the earthquake/wind/both but I finally got them both
05:18 Started MANUAL_INITIAL_ALIGNMENT
    05:24 Skipped INPUT_ALIGN_OFFLOADED due to same issues everyone else has had recently (76870, 76843)
        - I couldn't even get it to ACQUIRE_XARM_IR
05:41 Initial alignment complete, relocking
    - Having trouble locking ALSY due to high wind at EY - you can see the spot swinging on camera :(. Flashes look great though!
05:56 Going to sit in DOWN for 10 mins and rest the quads - wind just broke past 40mph
06:02 Tried to take advantage of a slight drop in wind speeds, but didn't work
06:16 Went back to sitting in DOWN
06:33 Trying locking again
06:35 Lockloss during LOCKING_ALS
06:48 Back to DOWN after multiple locklosses during LOCKING_ARMS_GREEN and LOCKING_ALS                                                                                                                                                                                                                                                                                                                                                                                                         

Ran the data for the In Lock SUS Charge Measurements that ran this morning. Like Camilla said (76895), the excitations only worked for the ETMs, so although I am attaching all four plots, note that the last plot point for the ITMs is March 13th (ETMX, ETMY, ITMX, ITMY).

We've been sitting in IDLE since the earthquake rolled through three hours ago. Ground motion is still too high to lock.

Functionality tests were performed on the LVEA turbo stations. The details:

X-manifold:
Bearings: 100%
Turbo hours: 1950
Scroll hours: 1946
The recently installed separated water lines were also tested and found water-tight.

Y-manifold:
Bearings: 100%
Turbo hours: 951
Scroll hours: 2279

OMC:
Bearings: 100%
Turbo hours: 5964
Scroll hours: 5903

Lockloss at 04/03 00:11UTC from Taiwanese earthquake that Jamie experienced in person

Last week Artem, Louis and I copied the new DARM filter into ITMX 76685. This week we got through the ETMX to ITMX ESD DARM transition but lost lock 12 seconds into the 20 second ITMX to ETMX transition.  Plot attached. The EX signal looks much noisier than IX, I need to check all filters are correct.

Again, as seen last week 76637, although both ETM excitations ran fine, neither ITM excitation ran. It's the same guardian code and logs that it's sending the excitations but they are not seen on the H1:SUS-ITMY_L3_LOCK_BIAS_EXCMON and H1:SUS-ITMX_L3_DRIVEALIGN_L2L_EXCMON channels. Will get help from cds team.

J. Kissel
ECRs E1700387 and E2400116
IIETs 9392 and 30863
WPs 11743 and 11797

With Dave's help we installed all of the front-end model changes described in LHO:76850, and I've followed up behind the model changes with upgrades to the MEDM screens to show the new infrastructure -- then filled in the infrastructure with EPICs values, settings, and filters.

So, as of now,
    - As with the rest of the BSC Quads, Triples, Doubles, and HAM Triples, the the HAM doubles and single suspensions now are ll using a fully functional, sensible BLRMS system, calibrated into microns_RMS.
    - The thresholds have been generically set to 150 [um_RMS] for starters, since we have little history of understanding how the SUS behave in this 0.1 to 10 Hz band in a physically meaningful way. Please tune as you see fit if it's troublesome at this starting value.
    - The OMC ASC that's fed to the OMC SUS is now routed through the SUS-OMC_M1_LOCK_[LPY] bank, now has proper 2k to 16k AI filtering in all of those DOFs, and the LOCK_[LPY] output goes through the DRIVEALIGN matrix.
    - However, in an effort to make the least changes to the existing functionality, I've turned OFF the P2L and Y2L gains that were in place but not used for the time being. Will commission the drivealign gains again later, but for now the OSC ASC path should behave exactly as it had before, just with a proper 2k to 16kHz AI filter that should not impact the control stability.
    - The channels that are stored in the frames that record the history of the control signal have changed 
        from H1:SUS-OMC_M1_ASC_[LTVRPY]_IN1_DQ
        to H1:SUS-OMC_M1_LOCK_[LPY]_IN1_DQ
        which includes H1:SUS-OMC_M1_ASC_[PY]_IN1_DQ which were (for some reason) 
        in the GDS Broadcaster frame list, so they should now be mapped one-to-one to 
        H1:SUS-OMC_M1_LOCK_[PY]_IN1_DQ. 
      @DetChar -- Maybe these were used in the summary pages or something? 
    - All model, MEDM, and filter changes have been committed to the userapps repo.
    - All new EPICs settings have been save in every impacted SUS's SDF system.

More details to come tomorrow.

TITLE: 04/02 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: SEISMON_ALERT
    Wind: 29mph Gusts, 24mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.20 μm/s
QUICK SUMMARY:

Currently relocking and in ENGAGE_ASC_FOR_FULL_IFO. Wind picking up and maxing out at 35mph

TITLE: 04/02 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

• Maintenance Day was fairly busy, but straightforward and mostly ended around noon.
• Unfortunately, Input_Align issues continue:  While locking red-X-arm, IMC gets a kick and loses lock.  PLUS if the Xarm does lock & is held at WFS Centering, the x-arm alignment immediately gets misaligned by the INPUT_ALIGN state.  (This has been an issue over the last few weeks.)
• H1 at the hand-off is now past CARM Offset Reduction (woo woo!)----this was by SKIPPING Input_Align part of Initial Alignment.
• In hindsight, probably should not have spent ~3hrs trying to figure out Input Align.  At any rate, Oli got it to NLN at the beginning of their shift.
• Ryan S also helped me heaps during the shift!

LOG:

• 1420 note of magnetic injections
• 1458 Lockloss
• 1512-1533 pmc align (ryanS)
  • 1533 then did rotation stage calib (ryanS)
• 1506 FARO set up and work (tyler)
• 1509 cable tray work in FC tube enclosure for non-invasive work (ken)
• 1517 Transitioing LVEA to SAFE (camilla)
• 1521-1625 Acme pest control escort around site (chris)
• 1525-1854 Purge air at end stations (gerardo)
• 1529-1554 Technical Cleaning at FCES (kim, karen)
  • 1555 EX (kim)
  • 1555-1646 EY (karen)
  • 1841 completed LVEA cleaning before maintenance ended (karen.kim)
• 1538 H1LSC guardian node RELOAD (camilla)
• 1555-1847 3ifo inventory checks + FARO (jason)
• 1601-1703 Beckhoff work (fernando)
• 1601-1703 Turbo pump test in lvea (janos)
  • 1743-1854 Back out to start spin down turbo & power off (janos)
• 1603-1858 Flipping sign for ETMx for 3hrs (rahul)
• 1618-1746 Maggie, new LHO Science educator, getting lvea  tour (Mike)
• 1624-1801 PSL down for beckhoff work (patrick.ryanS)
• 1626 FAMIS checks (chris)
• 1626-1839 DAQ restart related to model changes (dave.jeff.erik) 
  • Erik will try out an ndscope start script for after DAQ restarts ("restart_all_ndscopes" from terminal prompt runs it & it looked OK and restores within about 4-5min)
• 1637-1740 End Stations FAMIS work (mitch)
• 1653-1930 OMC DCPD Transimpednance Amp response meas at HAM6 racks (francisco.louis)
  • Misalign PRM, BS, ITMs, SR2 ?
  • 1910 Jeff joining
• 1710-1804 EX accelerometer checks + EY rack temperature investigation (marc)
• 1728 HAM7 ISI watchdog trip
• 1748-1835 Connect BRSX picomotor to EX picomotor chassis (fil, jim)
• 1748 Dropping sqz stuff off in lvea (camilla, eric)
• 1756-1943 SHG work in optics lab (julian)
• ~1800 Input Pointing work (jennie, camilla)
• 1831 Investigating missing channel in CER (marc, fernando)
• 1913-1930 LVEA sweep by Camilla (with OMC dcpd group still out on floor)
• 1925 Initial Alignment
  • Attempt #1 (right after Maintenance):  input align was having trouble.  IMC losing lock with lots of MC2 saturations.  Then after about 10min red Xarm locked, but Input Align drives it into misalignment and then the IMC breaks lock.  This was with a few different types of IM4 pit/yaw.
  • Attempt #2 (Tony's reversion):  After getting no where, decided to revert to Tony's gps time from Friday (see his alog #76483).  And then restarted the alignment (x & y green arms needed to be touched up by hand); then guardian took over for the green arms.  Unfortunately, the first attempt had the IMC break lock (as the red X-arm was acquiring).
  • Attempt #3 (Reversion to before last lock from yesterday):  Looking at another time to revert to (at 2123utc on 4/1/24,  which was about 24hrs ago right before the last lock when ISCLOCK was in the READY state).
    • GPS Time:  1396041841  (restore all except sqz)
    • Similarly to Attempt #2, X& Y green arms had to be touched up by hand (even more off on this attempt comparted to Attempt-2).
    • Once again, INPUT ALIGN knocked the IMC out of lock when trying to lock the red-X-arm.
  • Attempt #4 (Using #3, but now skipping Input Align!):  So, after ALIGN IFO went to DOWN-----> SKIPPED Input_Align step and continued with alignment.
    • PRC Align, Mich Bright each completed fine.
    • SRC needed to be helped by hand via SRM (mostly pitch adjustment).
    • Attempted locking after this.
• Lock#1:  Y-arm had decent flashes, but would not stay locked, and would be taken to several rounds of INCREASE FLASHES.
  • Since Increase Flashes was not making progress (would barely get Y-arm above 0.8); went to Green Arms Manual and touched up alignment by hand to around 1.0, and FINALLY made it to Locking ALS, but then there was an immediate lockloss.
  • ...and...Winds are picking up into the 30mph range. 
• Lock#2:  No Increase flashes needed for Locking Arms Green.  Went immediately to PRMI & locked.  DRMI locked.  Made it passed Prep ASC
• 2102-2121 Resetting EX HVAC chiller (eric)
• 2220 PCal Lab work (francisco)
• 2242 FC Enclosure to drop off parts (jordan)

WP 11799.

Patrick T., Ryan S.

The PSL Beckhoff PLC has been updated to match the changes made at LLO to add a watchdog channel. This is the code from 'TwinCAT Projekt_LIGO_240124.zip' in https://dcc.ligo.org/E2100235. The now running version is in git at https://git.ligo.org/cds/ifo/beckhoff/lho-psl and commit 9b6ad86cb26660b136242b334bd4831a683f1c5b. It appears there was also a change to fix a bug where toggling the noise eater tripped the NPRO. Ryan verified that this now appears to be fixed. I also removed the ${IFO}.PSL.LASER.MAIN.SHUTTER alias of the bEnable_Shutter2 variable, which according to LLO is not being used. The channel was not being exported to EPICS.

There were two minor issues. The first was that the version of a couple of the terminals found from a scan of the EtherCAT bus did not match what was in the IO configuration, and this appeared to be the cause of an INIT error for a couple of terminals (presumably those) when the system was put into run mode. This was fixed by changing the IO configured versions to match the scanned versions. The other was that I forgot to comment out the inversion of the signal from laser PD1, which was not realized until we started everything and were wondering why the power was reading as negative for that PD. I fixed that in a new commit and restarted the code again.

FAMIS 20022

The main chiller alarm was active last night, so this morning I added 100mL of water to stope the low-level alarm.

We plan on doing a remote PMC alignment tomorrow during maintenance to address the recent rise in PMC reflected power.