We lost lock 1.5 hours ago and have not been able to get back up due to an earthquake from Fiji rolling through. Currently sitting in DOWN until the ground motion slows

I set the camera_servo_offset_stepper.py script to run for CAM 1 servo between 10pm and 2am PDT and the CAM2  servo between 2am and 6am as the IFO unlocked at 6pm today when the first script was meant to run. Ths can run without knocking us out of OBS and has been cleared with Jenne.

I have by-hand re-engaged the ADS lines (and turned on the ITMY line that we don't usually turn on), so we can watch what they do during thermalization, while the spots are controlled with the camera servos. 

The lines have a bit been on-and-off, since we were going to NLN_CAL_MEAS, but in general the lines have been on at the nominal low noise height of 30 counts in the oscillators. 

The first plot is a zoom of what happened while we were naturally thermalizing, including some interesting-ness in the yaw3 value.  It's off and on again due to my not having it turned back on very quickly, but the values around -22 mins are what they had converged to when we switched to camera servos (and the lines get turned off).  Then, I turn the lines back on, and the yaw3 value is quite different from what it had been. I'm not sure if that's a huge deal or not, but it's something I hadn't really been actively aware of until today.

The second plot is over a much longer time, while Jennie and I were also moving some pointing around.  The cyan circles are of the time from the first plot.  The green-purple-green-blue horizontal bar in the middle is roughly representative of the times we were doing different things. Green times are normal, nominal thermalization, with all optics in their usual positions.  During the purple time Jennie was moving the PRM via the camera servo setpoint. During the second green time she had reverted things, so it's back to nominal thermalization.  The blue time is when I was moving IM1 and IM3 via their sliders.  

One conclusion I have is that the time whne we were moving IM1 and IM3 via their sliders (steps of -20 counts each in yaw, waiting, then another set of -20 slider counts each in yaw), the ADS YAW3 line goes farther from zero, indicating that we'd have higher ASC coupling, so it makes sense that the range might have looked worse when Gabriele and Jennie were doing this test a while ago, even though the buildups got better.  I don't think I had gone even quite as far in yaw as Jennie and Gabriele had gone the other day, but we had a fast lockloss, even though nothing was actively being moved (I was holding still for a few tens of seconds).

Jennie has more details on the tests we did this afternoon. 

F. Clara, M. Pirello, R. Schofield

We systematically verified the cabling from the VEA to the Electronics racks of all 7 accelerometers currently in service.  We replaced channel 5 on the new Accelerometer Power Conditioner for further testing, the new channel 5 is performing correclty.  Then we went through and tested the accelerometers vs the signals in NDSCOPE / DIAGGUI.  We found a fault in the EY_EBAY_Z cable and replaced it.  We corrected the cabling to match the system drawing D1300773 , while verifying the signals, we found an unlabeled cable that was not documented.  We determined that this cable was patch done before O4 to fix a bad accelerometer cable on BSC10_X, this cable was routed correclty and is now documented.  Final complete order of the cabling is as follows:

SignalName <= ADC# <= CBL# <= SLOT# <= AccelerometerName(path)
EY_OPLEV_X <= ADC7 <= CBL1 <= SLOT1 <= EY_OPLEV_X
EY_BSC10_X <= ADC8 <= CBL2 <= SLOT2 <= Black Patch Cable <= Patch Panel #3<= White Accelerometer Cable <= EY_BSC10_X
EY_BSC10_Y <= ADC9 <= CBL3 <= SLOT3 <= EY_BSC10_Y
EY_BSC10_Z <= ADC10 <= CBL4 <= SLOT4 <= EY_BSC10_Z
EY_TABLE_X <= ADC11 <= CBL5 <= SLOT5 <= EY_TABLE_X
EY_VEAFLOOR_Z <= ADC12 <= CBL6 <= SLOT6 <= EY_VEA_FLOOR
EY_EBAY_Z <= ADC13 <= CBL7 <= SLOT7 <= EY_EBAY_Z

The patched cable is a temporary solution, a permanent solution should be considered here.

TITLE: 03/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 19mph Gusts, 15mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.20 μm/s
QUICK SUMMARY:

Detector locked for 2 hours and currently commissioning.

TITLE: 03/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Maintenance day activities concluded just before noon and we made it back to low noise at 21:11 UTC. Since then there have been detchar injections, PEM inj, PRM/PR2 spot centering.

• First lock attempt
  • Ran an initial alignment, no issues
  • Made it up to Transition_to_ETMX before it lost lock. Violins were a bit high on ETMX, but can't say that this was the cause.
• 2nd
  • Waited at 2W DC to damp violins a bit before moving on

LOG:

Following instructions in this google doc and here's a past alog70817 to reference running as well.

I started the injections the first time at 1395523038, but ran into an HTTP error when trying to upload the event to GraceDb. I also had forgotten to turn off the PCALX lines and Robert had a shaker going on during part of that time as well. Scrap this one.

Second round with PCALX lines off, no shaker injections, and still ran into the HTTP error at the end of the injections. Start time of these - 1395523842.

Looking at the PCAL AOM, it seems that it has had a lower voltage for ~1 hour before these were ran. PCAL team is currently looking into this and if it would have affected these injections.

Naoki, Nutsinee

Since we see the 0.4 Hz motion in FC IR error signal, we added a boost filter for FC IR. The new boost filter is p0.1z5 and it is in FM9 of FC_LSC_DOF2. The first attachment shows the improvement of FC IR error signal with the boost filter. The rms is reduced by a factor of 3. We modified the SQZ_FC guardian to add this filter as follows.

line 393: ezca.switch('SQZ-FC_LSC_DOF2','FM1','OFF','FM2','OFF','FM3','OFF','FM9','OFF') #disable IR sus boost

line 424: ezca.get_LIGOFilter('SQZ-FC_LSC_DOF2').only_on('FM1','FM2','FM3','FM4','FM5','FM6','FM9','FM10','INPUT','OUTPUT','DECIMATION')

The SDF is accepted as shown in the second attachment.

I noticed in the summary pages spectrogram that the low frequency sensitivity was changing during one of the A2L step experiments (this is ETMX ITMX test).

The first plot shows the strain spectrum (GDS-CALIB_STRAIN_CLEAN) for 900 seconds for each available step of P2L and Y2L for the x arm. Some steps were repeated twice. In each panel, the title shows what A2L gain is being changed, and what's the nominal value. The second plot shows the ration of strain spectrum for a given A2L gain over the averaged strain spectrum with the nominal A2L value. So lower than one means the A2L value is improving the sensitivity.

There is evidence that yaw values away from nominla make the nosie below 30 Hz better.

TJ, Camilla, WP 11746, LLO did in LLO#14419

• Data set A: No mask. Razor blade 2 3/8" downstream of the center of the annular mask
• Data set B: Annular mask in mask. Razor blade 2 3/8" downstream of the center of the annular mask
• Data set C: No mask. Razor blade 10 7/8” upstream of the center of the annular mask

J. Kissel
ECR E1700387
IIET 9392
WP 11743

After 
    - yesterday's front-end code model prep (LHO:76696), 
    - TJ and I's prep for front-end code installation (LHO:76703), and
    - Dave's code installation (LHO:76706)
I spent the rest of the morning filling in the upgraded watchdog infrastructure for the HAM Triples, i.e. MC1, MC2, MC3, PRM, PR2, PR3, SRM, SR2, SR3, FC1, and FC1.

 All HAM Triples' WD RMS MAX thresholds have been set to 150 [um_RMS]
As with the QUADs and Beam Splitter -- please feel free to edit as we gain more experience with normal things that happen during IFO operation.

The following has been committed / was affected by today's upgrade.

MEDM SCREEN MODS
/opt/rtcds/userapps/release/sus/common/medm/hxts/
        SUS_CUST_HXTS_M1_WD_OSEMAC_RMSLP.adl
        SUS_CUST_HXTS_M2_WD_OSEMAC_RMSLP.adl
        SUS_CUST_HXTS_M3_WD_OSEMAC_RMSLP.adl
        
        SUS_CUST_HXTS_WD.adl
        
        SUS_CUST_HLTS_OVERVIEW.adl
        SUS_CUST_HSTS_OVERVIEW.adl

FOTON FILE MODS
/opt/rtcds/userapps/release/sus/h1/filterfiles/
        (all of the above mentioned foton filter files)

SDF FILE MODS
/opt/rtcds/userapps/release/sus/h1/burtfiles/
        (all relevant _safe.snap and _OBSERVE.snap files)

GUARDIAN MODS
/opt/rtcds/userapps/release/sus/common/scripts/
    sustools.py

Attached are my detailed notes.

This morning I took the OPLEV charge measurements on both ETMX and ETMY suspensions. I will post the results below after processing the data.

Both the suspensions were restored to their nominal state after the measurements were complete.

Later, Jeff flipped the sign on ETMX ESD bias voltage once the above measurements were done.

J. Kissel
ECR E1700387
IIET 9392
WP 11743

Tomorrow, we continue on the adventure towards upgrading the suspension watchdog systems that Oli, Dave and I have been chugging along on for the previous two weeks (see, eg, LHO aLOGs 76305, 76269 and 76545).
This week, we're tackling a big chunk -- all of the HAM triple suspensions, both large and smal, i.e. the 7 HSTS and 2 HLTS. 

Note :: these are the final suspensions that are hooked up to their respective ISIs, so this should be the last major upgrade.

As such I've:
    In the SUS model library parts,
    - Converted the BLRMS trigger generating systems to include a trust-worthy RMS calculation, coupled with a down-stream low-pass filter,
    - Removed all evidence of the USER DACKILL watchdog aggregator system
    - While I was there, I removed remaining evidence of the long-defunct / depricated "online detector characterization" ODC system, long since replaced by guardian state information
    On the top level SUS models,
    - Removed all evidence of the USER DACKILL watchdog aggregator system
    - Removed the IPC sender of the USER DACKILL to the respective ISI
    - Verified that the removal of library block output ports didn't botch any of the top level connections, reconnecting and re-organizing as needed
    On the top level of the ISI models,
    - Replaced the IPC receiver of the USER DACKILL watchdog from all respective SUS

which impacts all of the following models:

    Top Level ISI               Top Level Optic     Governing Library Part
    h1isiham2.mdl               h1susmc1.mdl        HSTS_MASTER.mdl
                                h1susmc3.mdl        HSTS_MASTER.mdl
                                h1suspr3.mdl        HLTS_MASTER.mdl
                                h1susprm.mdl        RC_MASTER.mdl

    h1isiham3.mdl               h1susmc2.mdl        MC_MASTER.mdl
                                h1suspr2.mdl        RC_MASTER.mdl

    h1isiham4.mdl               h1sussr2.mdl        RC_MASTER.mdl

    h1isiham5.mdl               h1sussr3.mdl        HLTS_MASTER.mdl
                                h1sussrm.mdl        RC_MASTER.mdl

    (no change to isiham7)      h1susfc1.mdl        HSTS_MASTER.mdl

    (no change to isiham8)      h1susfc2.mdl        HSTS_MASTER.mdl

In addition, some of the lower level SUS library parts were also impacted,
    SIXOSEM_T_STAGE_MASTER.mdl
    FOUROSEM_STAGE_MASTER.mdl
    FOUROSEM_STAGE_MASTER_OPLEV.mdl

All of the above model changes have been committed to the userapps SVN repo in revs 27310, 27311, 27312, and 27313.
    

We aimed to do A2L steps over the weekend, 15 minutes/step  76630, 76683, but as Elenna points out with the camera servos running the beam spot stayed static on the mirror while the mirror actuation point changed.

Looking at data from A2L tuning in April 2023 68384 attached plot, A2L steps are ~0.4 for a change of around 2 in the camera offset  (TRAMP 120s). We'll keep around the steps same as the weekend: +/- 1.0 per 15 minutes to up to +/-2.0 in the H1:ASC-CAM_{PIT,YAW}{1,2,3}_OFFSET channels during observing with nominal TRAMPs extended from 10s to 120s. These offsets are already unmonitered in sdf as reset each lock, I unmonitered the TRAMPs.

These offfsets vary up to 3 counts lock to lock, see attached plot. Set based on ADS converging.

Script saved in /ligo/gitcommon/labutils/beam_spot_raster/camera_servo_offset_stepper.py  Scheduled to run if we're in NLN via tmux session on cdsws26 at:

• 6pm - 10pm on CAM1 (BS => CAM1 => PRM)
• 10pm to 2am on CAM2 (ETMX => CAM2 => ITMX, ETMX)
• 2am to 6am  on CAM3 (ETMY => CAM3 => ITMY, ETMY)

Tony will be here during the first set so can cancel the others if there is any issues.

As shown in the attachment, I turned on the beam spot control. The flag of the beam spot control in sqzparams is set to True. The beam spot control seems working well, but I feel the yaw was too slow so I removed the -20dB gain (FM7 in INJ_ANG_Y). I will tune the green QPD offset with FC2 dither tomorrow.

These changes were accpted, but need to be set again after aligning the FC IR transmission.

Jason, Filiberto, Fernando.

Per the WP11763

Motor/Brake cable identification was perfromed for the TIMX/ITMY/BS OPLevers. The provisional Medm was tested as well. We took advantage of this opportunity to complete the internal wiring in the BS OPLEver with the inclusion of the RJ9 patches.

The WP will remain open to perform the activity in ETMX/ETMY next Tuesday. 

Evan, Naoki, Nutsinee, Camilla

Today Sheila added nodes.set_managed() in INIT state of SQZ_FC guardian and let SQZ_FC guardian go through the INIT state. However, the FC2 M1 offload filter in the INIT state was old filter setting in 68914, which caused GR_SUS_LOCKING failure. We reverted it as shown in the attachment and updated the FC2 M1 offload filter in INIT state.