TITLE: 08/26 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 139Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: The shift was mostly taken up by commissioning time. There was recovery from a CDS crash this morning, but recovery was pretty straight forward. The reacqusitions from the two lock losses one needed an initial alignment, the other did not. Minor interventions for both.
LOG:

• 1744 Back to low noise, time for commissoining
• 1827 Lock loss
  • Tried relocking without an initial alignment, didn't work. Fully auto IA and main locking
• 2059 Observing
• 2159 Lock loss
  • No initial alignment, but I held at Locking_Arms_Green because it would move to the next state before the WFS would converge and cause a lock loss. We then lost lock at Engage_ASC_For_Full_IFO, and the next time it made it up on its own.                             

I restarted the 4 SUS in lock charge nodes and the PEM magnetic injection node to reestablish the connection to the front ends that can go away after front end crashes, like the one this morning ( alog79708). We wanted to be prepared for these to run tomorrow morning if the IFO is locked.

FAMIS 27796

• TCSx chiller: Level of 29.8; did not add water
• TCSy chiller: Level of 9.8; did not add water
• No water seen in small cup under the "slow leak"
• Values updated in T2200289

Before the vent, we had lowered the DARM offset used at the end of the DARM_OFFSET state for locking the OMC since we had seen the PRG fall off and cause a lockloss with the nominal offset of 9e-05 (see alog79082 for details). When locking this afternoon, I raised the offset from 6e-05 back to 9e-05 after running through DARM_OFFSET, and seeing that the PRG didn't plummet and cause a lockloss, we continued locking. The OMC locked on the first try, something that hasn't been the case recently, so having more carrier light there seems to help. I compared OMC scans from this lock against the last lock, which used the lower DARM offset; attachment 1 shows the scan with the higher offset and attachment 2 with the lower offset. According to the OMC-DCPD_SUM channel, we get ~10mW more carrier light on the OMC when locking with this higher DARM offset.

I've put this DARM offset of 9e-05 back into ISC_LOCK and loaded it. We can watch over the next couple of lock acquisitions to see if the problem wth the PRG dropping off resurfaces.

I measured OMC fringe wrapping after OFI replacement. First I ran the dtt template in 78942, which caused a lockloss. In the next try, I reduced the excitation from 3421 to 600, which corresponds to 2.4 um pp OMC L (H1:SUS-OMC_M1_DAMP_L_IN1_DQ). The attachment shows that OMC scatter is worse than one in April (blue: April vs purple: today). We may need to adjust the OFI temperature.

FAMIS 26290


Laser Status:
    NPRO output power is 1.831W (nominal ~2W)
    AMP1 output power is 64.55W (nominal ~70W)
    AMP2 output power is 137.1W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 6 days, 23 hr 49 minutes
    Reflected power = 21.13W
    Transmitted power = 105.6W
    PowerSum = 126.7W

FSS:
    It has been locked for 0 days 1 hr and 37 min
    TPD[V] = 0.6479V

ISS:
    The diffracted power is around 1.9%
    Last saturation event was 0 days 1 hours and 37 minutes ago

Possible Issues:
    AMP1 power is low
    PMC reflected power is high
    FSS TPD is low
    ISS diffracted power is low

FAMIS 21189

Nothing much of note this week. PMC REFL is still increasing slowly while PMC TRANS is decreasing.

The FSS TPD signal is still low, and since I wasn't able to increase it much last week, we plan to go into the enclosure and tune up the FSS path on-table soon to fix it.

There were a few PCAL SDF diffs when we made it back to low noise today at CALEY. It looked like these were loaded into epics from teh safe.snap and these numbers disagreed with what was saved in the observe.snap file. I confirmed with Franscisco that this was the case, and he had me verify that they agreed with alog77386. I then also saved these new values in the safe.snap file as well. Interestingly, these channels are not monitored in the safe.snap.

I ran A2L while we were still thermalizing, might run again later. No change for ETMY but the ITMs had large changes. I've accepted these in SDF, I reverted the tramps that the picture shows I accepted. I didn't notice much of a change in DARM or on the DARM blrms.

         ETMX P
Initial:  3.12
Final:    3.13
Diff:     0.01

         ETMX Y
Initial:  4.79
Final:    4.87
Diff:     0.08

         ETMY P
Initial:  4.48
Final:    4.48
Diff:     0.0

         ETMY Y
Initial:  1.13
Final:    1.13
Diff:     0.0

         ITMX P
Initial:  -1.07
Final:    -0.98
Diff:     0.09

         ITMX Y
Initial:  2.72
Final:    2.87
Diff:     0.15

         ITMY P
Initial:  -0.47
Final:    -0.37
Diff:     0.1

         ITMY Y
Initial:  -2.3
Final:    -2.48
Diff:     -0.18
 

Mon Aug 26 08:02:15 2024 INFO: Fill completed in 2min 13secs

Short but a good fill. Gerardo is reducing the LLCV.

Jordan cleared an ice buildup at the end of the discharge pipe.

TJ, Jonathan, EJ, Dave:

Around 01:30 this morning we had a Dolphin crash of all the frontends at EY (h1susey, h1seiey, h1iscey). h1susauxey is not on the Dolphin network as was not impacted.

We could not ping these machines, but were able to get some diagnostics from their IPMI management ports.

At 07:57 we powered down h1[sus,sei,isc]ey for about a minute and then powered them back on.

We checked the IX Dolphin switch at EY was responsive on the network.

All the systems came back with no issues. SWWD and model WDs were cleared. TJ is recovering H1.

TITLE: 08/26 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 0mph Gusts, 0mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY: Looks like we lost the SUSEY and ISCEY front ends. This created connection errors and put many Guardians, including IFO_NOTIFY, into an error state. Contacting CDS team now.

TITLE: 08/24 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Have had the BS camera die on us a couple times in the last 24hrs (this requires contacting Dave to restart the camera's computer--atleast until this camera can be moved to another computer), and so the beginning of the shift was restoring this and also figuring out why H1 had a lockloss, BUT did not drop the Input Power down from 61W to 2W.

After the camera issue was fixed, ran an alignment with no issues, and then took H1 all the back to NLN also with no issues.  

Then completed taking Calibration measurements after 1.5hrs and 3hrs.  Oli also ran LSC Feedforward measurements......

Then there was ANOTHER BS Camera Computer Crash!  Dave brought us back fast.  

Now back to OBSERVING!
LOG:

• Arrived to see that H1 was stuck at ADS TO CAMERAS, BUT had had a lockloss.  ALSO: Input Power did not change after the lockloss---so, 61W was being dumped to MC REFL!  Mystery as to why this happened.  I took H1 DOWN and this took care of the High Power onto MC REFL.  
• 1611-1641 INITIAL ALIGNMENT
  • x+y green arms were way out and needed INCREASE FLASHES
  • Everything else went fine.
• 1635 Noise hunting in Mechanical Room off and on today (robert)
• 1643-1728 Locking H1 & then getting stuck at ADS TO CAMERAS (same as last night)---Noticed that the CAMERA SERVO had stalled and noted issues with CAM1 Pit & Yaw.  Posted this onto LIGO Chat and Ryan Short mentioned it's probably due to the BS' camera.  Sure enough it had died!
• 1800-1807 Contacting Dave to restart the BS camera
• Save & Loaded LASER_PWR.py  (It had already been edited for the change to 61W.)
• 1812 NLN for H1
• 1946-2015 Ran Calibration after H1 was locked for 90min.
• 2042 H1 back to OBSERVING (1st time since July12th!)
• 2113-2301 Out of Observing:  Running Thermalized Calibration + LSC Feedforward measurements
  • Since the BS Camera happen to die during Oli's last LSC FF measurement for PRCL, they ran it again.
• 2226 - 2243 BS Camera was down again!  (Dave brought it back for us pretty fast.)
• 2301 OBSERVING

We can edit this list as needed. 

We are having trouble locking this afternoon because of the wind, but we have some requests for the operators this weekend if they are able to relock. 

We've changed the nominal locking power to 61W, in the hopes that this might avoid the PI or let us pass through it quickly.

When we first get to NLN, please take the IFO to NLN_CAL_MEAS and run a calibration sweep.  If we stayed locked for ~1.5 hours, please re-run, and if we are ever locked for more than 3 hours please re-run again. 

After the calibration has run, we would like to check the camera set points since we have seen earlier today that they are not optimal and that might be related to our PI problem.  We already updated the offset for PIT1, but we'd like to check the others.   We modified the script from 76695 to engage +20 dB filters in all these servos to speed up the process.   Each DOF should take a little more than 15 minutes.  We'd like these run with the POP beam diverter open, so we can see the impact on POP18 and POP90.  The operators can run this by going to /ligo/gitcommon/labutils/beam_spot_raster and typing python camera_servo_offset_stepper.py 1 -s now  (and once 1 completes, run 2 and 3 if there's time.)

I've added 3 ndscope templates that you can use to watch what these scripts do.  the templates are in userapps/asc/h1/templates/ndscope/move_camera_offsets_{BS, DOF2_ETMX, DOF3_ETMY}.yml  We'd like to see if any of these can increase the arm powers, or POP18.  If there is a better value found, it can be set to the default by updating lscparams.py lines 457 to 465. 

R. Short, F. Clara

In the ongoing effort to mitigate the 9.5Hz comb recently found to be coming from the PSL flow meters (alog79533), this afternoon Fil put the PSL control box in the LVEA PSL racks on its own, separate 24V bench-top power supply. Once I shut down the PSL in a controlled manner (in order of ISS, FSS, PMC, AMP2, AMP1, NPRO, chiller), Fil switched CB1 to the separate power supply, then I brought the system back up without any issues. I'm leaving the ISS off for now while the system warms back up. I'm leaving WP12051 open for now until enough data can be collected to say whether this separate supply helps the 9.5Hz comb or not.

Please be aware of the new power supply cable running from under the table outside the PSL enclosure to behind the racks; Fil placed a cone here to warn of the potential trip hazard.