TITLE: 08/24 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Have had the BS camera die on us a couple times in the last 24hrs (this requires contacting Dave to restart the camera's computer--atleast until this camera can be moved to another computer), and so the beginning of the shift was restoring this and also figuring out why H1 had a lockloss, BUT did not drop the Input Power down from 61W to 2W.

After the camera issue was fixed, ran an alignment with no issues, and then took H1 all the back to NLN also with no issues.  

Then completed taking Calibration measurements after 1.5hrs and 3hrs.  Oli also ran LSC Feedforward measurements......

Then there was ANOTHER BS Camera Computer Crash!  Dave brought us back fast.  

Now back to OBSERVING!
LOG:

• Arrived to see that H1 was stuck at ADS TO CAMERAS, BUT had had a lockloss.  ALSO: Input Power did not change after the lockloss---so, 61W was being dumped to MC REFL!  Mystery as to why this happened.  I took H1 DOWN and this took care of the High Power onto MC REFL.  
• 1611-1641 INITIAL ALIGNMENT
  • x+y green arms were way out and needed INCREASE FLASHES
  • Everything else went fine.
• 1635 Noise hunting in Mechanical Room off and on today (robert)
• 1643-1728 Locking H1 & then getting stuck at ADS TO CAMERAS (same as last night)---Noticed that the CAMERA SERVO had stalled and noted issues with CAM1 Pit & Yaw.  Posted this onto LIGO Chat and Ryan Short mentioned it's probably due to the BS' camera.  Sure enough it had died!
• 1800-1807 Contacting Dave to restart the BS camera
• Save & Loaded LASER_PWR.py  (It had already been edited for the change to 61W.)
• 1812 NLN for H1
• 1946-2015 Ran Calibration after H1 was locked for 90min.
• 2042 H1 back to OBSERVING (1st time since July12th!)
• 2113-2301 Out of Observing:  Running Thermalized Calibration + LSC Feedforward measurements
  • Since the BS Camera happen to die during Oli's last LSC FF measurement for PRCL, they ran it again.
• 2226 - 2243 BS Camera was down again!  (Dave brought it back for us pretty fast.)
• 2301 OBSERVING

3.5 hours into our lock we took LSCFF measurements. Files can be found in /opt/rtcds/userapps/release/lsc/h1/scripts/feedforward/ and are basically the name of the measurement template+'_20240824', but I'll list their filenames below anyway.

MICH Preshaping (run as 30 fixed averages)
MICHFF_excitation_ETMYpum_20240824.xml
MICHFF (30 accumulative averages)
MICH_excitation_20240824.xml

SRCL Preshaping (32 accumulative averages)
SRCLFF_excitation_ETMYpum_20240824.xml
SRCLFF (30 accumulative averages)
SRCL_excitation_20240824.xml

PRCLFF (30 accumulative averages)
PRCL_excitation_20240824.xml

For H1 recovery road back to Observing, it was requested to take calibrations (1) immediately at NLN, (2) 1.5hrs after NLN, and then the (3) standard one of atleast 3hrs after NLN.

Measurement NOTES:

• Following instructions (link)
• H1 at NLN for atleast 1.5hrs.
• Calibration_Monitor medm screenshot is attached
• 2113 Out of Observing, ISC LOCK to NLN_CAL_MEAS & Observatory Mode to CALIBRATION
• 2114- 2119  Broadband measurement run
  • Measurement at:  /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/
• 2121-2144 Simullines measurement
  • gps START time:  GPS: 1408569700
  • gps END time:  GPS: 1408571082.513554
  • Files writtten to subfolders here:  /ligo/groups/cal/H1/measurements

Now that this is complete, we can cross this off the white board, and Oli immediately started running LSC Feed Forward measurements.

(Corey, Oli, RyanS [remote], Sheila [remote])

After roughly 6-weeks (Jul 12, 2024), H1 made it back to Observing (at 2042utc Aug24) to join L1 & V1 (albeit briefly since we needed to drop from Observing to take a thermalized calibration measurement + LSC FeedForward offset measurement).  H1 range is only about 140Mpc.

We are running with input power setting of 61W (to help avoide PI Mode24). 

To get to Observing, SDF Diffs needed to be ACCEPTED (see attached image with all the diffs).

For H1 recovery road back to Observing, it was requested to take calibrations (1) immediately at NLN, (2) 1.5hrs after NLN, and then the (3) standard one of atleast 3hrs after NLN

Measurement NOTES:

• Following instructions (link)
• H1 at NLN for atleast 1.5hrs.
• Calibration_Monitor medm screenshot is attached
• 1946-  ISC LOCK to NLN_CAL_MEAS & Observatory Mode to CALIBRATION
• 1946-1951 Broadband measurement run
  • Measurement at:  /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/
• 1952-2015 Simullines measurement
  • gps START time:  GPS: 1408564372
  • gps END time:  GPS: 1408565741.455985
  • Files writtten to subfolders here:  /ligo/groups/cal/H1/measurements

h1cam26 (BS) was restarted Thur 12:56 PDT after stopping late Wed 22:23. It only lasted less than two days and stopped responding Fri night at 22:30.

I restarted it again this morning at 11:07 following the standard procedure.

H1 is currently stuck at ADS TO CAMERAS (same as last night); see attached screenshot.

CAMERA SERVO node has messages of:

"ASC-CAM_PIT1 (& YAW1)_INMON is stuck!  Do Not Advance!"

Will do some searching to see what the issue is. 

NOTE:  Sheila's alog that there was mention of PIT1 being updated.

Arrived to see H1's ISC LOCK at ADS TO CAMERAS, but no signals on the front ndscope and no DARM line on DARM FOM.  SUS  Violin screen looked like everyone was OFF or not damping.   Eventually, the telltasign was MC Trans was flashing (so IMC was NOT locked) and MC REFL was really bright (with OPS Overview saying the PSL was at ~61W.

OK, here's a rough timeline for last night (used the handy "guardctrl log" command line command which Camilla was able to help me find; I also attached the guardtrl log for all notes for the minute around the lockloss):

• 537utc (1037pmlocal):  ISC_LOCK at ADS TO CAMERAS
• 0718 Verbal Alarms:  "H1 Assistance Required"
  • Not really sure what this was about....I'm guessing its related to Automatic operations because H1 MANAGER guardian was STALLED and mentioned the "assistance required"
• 09:10:45utc On Verbal Alarms we get saturations for many SUS and PDs
  • (this is probably when the lockloss occured)
  • Fast Shutter closes (Camilla mentioned this, so I immediately checked that.)
  • IMC loses lock
  • PSL Power remains 61 Watts
• 1453 GRD IFO taken from AUTO to MANAGED
• 1500 ISC LOCK eventually taken DOWN and everything looks somewhat normal.

I will probably run an Initial Alignment and return to the game plan of locking H1 and running measurements at NLN.

Sat Aug 24 08:13:53 2024 INFO: Fill completed in 13min 49secs

TITLE: 08/24 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: n/a
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:

Trying to figure out the state of H1, it was in automatic operation and is currently stuck at ADS_TO_CAMERAS since before 11pm local.  (My main confusion is why I don't see any signals on nuc 30 (no DARM signal and no signals on the PRMI.sb ndscope.  I also don't see Odd.)

WAIT. 

OK, H1 is DOWN, but Guaridian seems to think it's trying to get thru ADS to Cameras and Inject Squeezing.....my realization is that MC Trans was flashing!  Then I saw that MC Refl is really bright (I'm assuming close to the 60.8W Guaridan had taken us to via ISC LOCK.

TITLE: 08/24 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY: Currently relocking and at MOVE_SPOTS. Max power is currently 61W(in actuality we are getting 60.8W out) in lsc_params in the hopes of avoiding the new PI. High winds today made locking hard, but we finally were able to get into NLN for an hour, and we were able to take (unthermalized) calibration measurements(79676), as well as figure out where the new PI is from and started trying to update the camera offsets. While trying to run the script that checks for the best camera offset locations, we lost lock, and I have been trying to get us relocked since then. We kept having locklosses in the middle states (~200-420s), and this last time Sheila recommeded fixing alignment by sitting in OFFLOAD_DRMI_ASC and adjusting SRM and PRM to raise POP18 and lower POP90 (altough I mostly focused on raising POP18), and adjusting it seems to have helped us get part this trouble area. Also, OMC_LOCK has been having a little trouble locking the cavity, but it will eventually catch on its own. For tonight, I am setting H1_MANAGER to LOW_NOISE, but leaving IFO_NOTIFY off.
LOG:

23:30 Just lost lock from LOWNOISE_ASC
00:59 NOMINAL_LOW_NOISE
01:06 Went to NLN_CAL_MEAS to run calibration
01:35 Calibration measurements finished, went back to NLN_CAL_MEAS
02:00 Lockloss
    02:13 Lockloss from ACQUIRE_DRMI_1F
    02:24 Lockloss from CHECK_MICH_FRINGES
    02:26 Started an initial alignment
    03:05 Initial alignment done, relocking
    03:20 Lockloss from RESONANCE
    03:50 Lockloss from CARM_150_PICOMETERS
    04:01 Lockloss from RESONANCE
    04:13 Lockloss from CARM_TO_REFL
    04:22 Lockloss from ENGAGE_DRMI_ASC    
        - Starting an initial alignment
    04:41 Initial alignment done, relocking
    04:53-04:58 Sat in OFFLOAD_DRMI_ASC and adjusted SRM and PRM to raise POP18

** These calibration measurements were started a few minutes after reaching NLN. The IFO was NOT thermalized!!

Calibration monitor screenshot

Broadband (2024-08/24 01:06 - 01:11 UTC)

File:

/ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240824T010639Z.xml
 

Simulines (2024/08/24 01:13 - 01:35 UTC)

Screenshot of simulines error from the end of the measurement

Files:

/ligo/groups/cal/H1/measurements/DARMOLG_SS/DARMOLG_SS_20240824T011302Z.hdf5
/ligo/groups/cal/H1/measurements/PCALY2DARM_SS/PCALY2DARM_SS_20240824T011302Z.hdf5
/ligo/groups/cal/H1/measurements/SUSETMX_L1_SS/SUSETMX_L1_SS_20240824T011302Z.hdf5
/ligo/groups/cal/H1/measurements/SUSETMX_L2_SS/SUSETMX_L2_SS_20240824T011302Z.hdf5
/ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_20240824T011302Z.hdf5

Reconciling some ASC observe diffs. Attached screenshot shows the change.

No SRC1 offset anymore, new POP A offsets, and a change in the tramp for the WFS offset.

There are still diffs for the ADS LO PIT and YAW matrix. I don't know why that is so I am not changing them.

After reaching NLN this morning, I accepted some SDFs for ALSE{X,Y} new green references and WFS PZT biases - attachments 1 and 2

We can edit this list as needed. 

We are having trouble locking this afternoon because of the wind, but we have some requests for the operators this weekend if they are able to relock. 

We've changed the nominal locking power to 61W, in the hopes that this might avoid the PI or let us pass through it quickly.

When we first get to NLN, please take the IFO to NLN_CAL_MEAS and run a calibration sweep.  If we stayed locked for ~1.5 hours, please re-run, and if we are ever locked for more than 3 hours please re-run again. 

After the calibration has run, we would like to check the camera set points since we have seen earlier today that they are not optimal and that might be related to our PI problem.  We already updated the offset for PIT1, but we'd like to check the others.   We modified the script from 76695 to engage +20 dB filters in all these servos to speed up the process.   Each DOF should take a little more than 15 minutes.  We'd like these run with the POP beam diverter open, so we can see the impact on POP18 and POP90.  The operators can run this by going to /ligo/gitcommon/labutils/beam_spot_raster and typing python camera_servo_offset_stepper.py 1 -s now  (and once 1 completes, run 2 and 3 if there's time.)

I've added 3 ndscope templates that you can use to watch what these scripts do.  the templates are in userapps/asc/h1/templates/ndscope/move_camera_offsets_{BS, DOF2_ETMX, DOF3_ETMY}.yml  We'd like to see if any of these can increase the arm powers, or POP18.  If there is a better value found, it can be set to the default by updating lscparams.py lines 457 to 465.