Here are the GPS times we've spent in the NEW DARM state. This is primarily useful for Keita, Sheila, and I as we try to better understand why weren't able to stay in the state in early Jan.

Time based on Sheila's investigative work from earlier this month: LHO:75308 and LHO:75348. The table of NEW DARM periods below does not include failed attempts. Instead, I just included successful transitions to the NEW DARM state (GRD STATE 710) in hopes of following up on LHO:75432. In addition, I've check the CAL-CS filterbank and gain states for each of the NEW DARM transitions in the table below as well. All but the last entry contain at least 10 minutes of "undisturbed" time during which CAL-CS was not changed. During the last window (GPS 1387237749-1387239372), CAL-CS saw many changes. 




  
1386536646
  
1386544194
  
02:06:09
  
more info in LHO:75348; no CAL-CS changes during stretch


  
1386619903
  
1386623884
  
00:35:41
  
more info in LHO:75348; no CAL-CS changes during stretch


  
1387033311
  
1387033995
  
00:11:10
  
successful transition, more info in LHO:75348; no CAL-CS changes during stretch


  
1387140065
  
1387141308
  
00:20:51
  
more info in LHO:75348; no CAL-CS changes during stretch


  
1387237749
  
1387239372
  
00:26:49
  
more info in LHO:75348, initially discussed in LHO:74977 CAL-CS changed throughout. see NEW_DARM_scope_1387237749.png





== 
I used the ndscope command below to check the CAL-CS pipeline for changes:
ndscope H1:GRD-ISC_LOCK_STATE_N . H1:CAL-CS_DARM_FE_ETMX_L1_LOCK_L_SWSTAT , H1:CAL-CS_DARM_FE_ETMX_L1_DRIVEALIGN_L2L_SWSTAT , H1:CAL-CS_DARM_FE_ETMX_L1_LOCK_L_GAIN H1:CAL-CS_DARM_FE_ETMX_L1_DRIVEALIGN_L2L_GAIN . H1:CAL-CS_DARM_FE_ETMX_L2_LOCK_L_SWSTAT , H1:CAL-CS_DARM_FE_ETMX_L2_DRIVEALIGN_L2L_SWSTAT , H1:CAL-CS_DARM_FE_ETMX_L2_LOCK_L_GAIN H1:CAL-CS_DARM_FE_ETMX_L2_DRIVEALIGN_L2L_GAIN . H1:CAL-CS_DARM_FE_ETMX_L3_LOCK_L_SWSTAT , H1:CAL-CS_DARM_FE_ETMX_L3_DRIVEALIGN_L2L_SWSTAT , H1:CAL-CS_DARM_FE_ETMX_L3_LOCK_L_GAIN H1:CAL-CS_DARM_FE_ETMX_L3_DRIVEALIGN_L2L_GAIN . H1:LSC-DARM1_SWSTAT , H1:LSC-DARM1_GAIN

In LHO:75560, Dana found that cal report 20230505T200419Z was not thermalized. As such, it should not be included in the GPR and uncertainty budget calculations. I've removed the valid tag in that report.

As soon as the VAC team removed the HAM3 West door, Betsy and I went in to lock the ISI. I locked them in reverse order (D,C,B,A) due to access, climbing into the beamtube to get to B and A.

This work is following up on LHO:73735. 

I've written a script that retroactively fixes the pyDARM parameter issues discussed in LHO:73735. The script lives here on the CDS workstations: /ligo/home/louis.dartez/projects/20240124_script_fix_bad_inis_from_alog_LHO73735/fix_bad_site_inis.py.

Since the pyDARM parameter INI files were initially written in error, anyone trying to process old corresponding to the affected reports' times would be using the wrong IFO models. As such, I've used the script above to fix the affected reports in situ. The original reports have been copied into /ligo/groups/cal/H1/reports/archive/reports_preserved_from_fix_for_LHO73735.

WP11646 New h1sqz model

Daniel, Dave:

A new h1sqz model was installed. DAQ restart was required

WP11651 Add New SQZ_PMC Guardian nodel to DAQ

Vicky, Camilla, Daniel, Dave:

The new SQZ_PMC GRD node was added to H1EPICS_GRD.ini. DAQ + EDC restart required

DAQ Restart

Dave:

The DAQ was restarted for the above changes. Sequence was 0-leg, EDC, 1-leg.

No major problems with the restart, both GDS daqds had to be restarted a second time for channel list synchronization.

TITLE: 01/30 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: Tony
INCOMING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 5mph 5min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.46 μm/s
QUICK SUMMARY:

Revovering from the Dolphin timing crash desribed in Alog 75612
We have taken all the SUS pensions to safe as soon as I could.
Dave restarted all the IOPs.
Rahul Recovered all the SUS from Safe
The CDS issues seem to be resolved by 17:28 UTC

Dainel started working on Beckhoff restarts, Ran into issues at EY.

SQZ Model was restarted which did effect FW0 & FW1 so there is a breif time of no data being written to the frames.

Moving Forklift in the LVEA

18:00 UTC Beckhoff slow controls just went down across the Site including the PSL.
18:04 UTC Beckoff back online.

 
PSL_ENV_LASERRM_ACN_TEMP_DEGF alarm 18:33 UTC
Trended the channel, looks fine.

Beam Splitter ISI tripped.
Beam Splitter ISI Reset.
Beam Splitter SUS is in Damped.

The LVEA is Now LASER SAFE for the HAM3 Door removal

day 1 (alog 75548), day 2 (alog 75557), day 3 (alog 75575), day 4 (alog 75601)

Recovery from reboots

After some things were rebooted, we've found that the suspension slider offsets for all OMs, SRM and ZM5 were reverted back to old-ish numbers. I and Camilla manually restored that.

Camilla saw that the beam was not quite back to the old position on one of the HAM7 QPDs but wasn't that bad.

The beam was already on ASC-AS_C but not centered, so I centered it using SRM.

Following that, I had to make a minor tweaking of OM1/2/3 to recenter OMC qpds quickly.

HAM6 irises are centered

Preet and Sheila recentered the two irises on HAM6. From this point on, these irises are a fiducial for the IFO beam.

OMC trans video beam and OMCR beam dump will be done later this week

LVEA was transitioned to laser safe for HAM3 door removal. We'll continue suspension work in HAM6.

Jonathan, Erik, Tony, Rahul, Richard, Dave:

Last night we had another one of the corner station glitches which took down all the models on h1susb123, h1sush2a, h1sush34 and h1sush56.

The glitch happened at 18:45:55 PST Mon 29th Jan 2024 during the O4a,b break.

This has happened several times before, details in FRS30324, with a regularity of every 80 to 100 days.

After the glitch, the dolphin IPC continued to function, which meant that the SWWDs did not trip the associated Seismic systems.

DMESG logs did not show anything.

I ran a complete set of Dolphin DIS_DIAGS (previous run was 24th Jan 2024), which only showed the issues with OAF0 and SEIH16 from last Friday's crashes. No problems with any SUS was seen.

Recovery from the crash was:

• Set all SEI IOP SWWD system into BYPASS mode with a very long bypass time (999999999S)
• Restart all the models on h1sus[b123, h2a, h34, h56]. All started with no problems.
• Unbypass the SEI SWWDs.
• Hand over to control room for SUS recovery.

The H1SQZ model has been updated to fix the PMC_TRANS & FIBR_PD readbacks, and to add fast DQ channels for the new PMC readbacks.

The EtherCAT system was been updated to fix the PMC_REFL PD type.

A new squeezer guardian node has been added for the PMC which required a DAQ restart to add the new channels.

Tue Jan 30 10:12:06 2024 INFO: Fill completed in 12min 2secs

Jordan confirmed a good fill curbside, he removed some ice buildup around the discharge line vent.

There was a timing glitch that affected many models running on corner station front-ends, and the IOPs on h1susb123, h1sush34, h1sush2a, h1sush56.

The timing error is on the order of tens of milliseconds, which is consistent with a Dolphin glitch.  Non-Dolphin front-ends and end station front ends were not affected.

Error was at about Jan 30, 02:45 UTC.  Models are running, but the affected IOPs will need to be restarted.

day 1 (alog 75548), day 2 (alog 75557), day 3 (alog 75575)

HAM7 irises were good

Sheila/Camilla checked the iris position on HAM7 and it was good.

ASC-AS_C whitening gain was increased by 18dB, dark offset was reset

I didn't like that the ASC-AS_C input was so small. Increased the whitening gain by 18dB (from nominal 18dB to 36dB) and reset the dark offset.

Recentered the beam on ASC-AS_C. One thing that was strange was that the ASC-AS_C_NSUM would become MUCH bigger (like a factor of 10) when SRM is misaligned. I was worried that I was looking at a ghost beam. Camilla measured the  beam power to be ~1mW out of HAM7 and ~0.7mW into HAM6. When ASC-AS_C was centered, ASC-AS_C_NSUM_OUT became ~0.008 give or take some. Taking the 16dB extra whitening (i.e. a factor of 8) into account, ASC-AS_C_NSUM~0.008 means about 1mW into HAM6, which was in a ballpark, so I convinced myself that the beam on AS_C was good.

HAM6 irises and beam height on OM, the beam was still very low on OM2

At this point OMC QPDs are reasonably centered, so Sheila and Camilla checked the beam position on irises in HAM6.

The beam was OK on the first iris but was a bit low (~2mm) on the iris closer to the OM1.

The beam position on OMs at this point as well as the slider values and max DAC output are listed below (see Camilla's pictures, too). Note that the YAW position in the table is the position of the incoming beam on the mirror measured at some (unknown) distance, it's not on the mirror.  

The beam was clearing the input and output hole on the shroud, was cleanly hitting the small OMCR steering mirror by the cage, and was already going to the OMCR diode.

They confirmed that the OMC trans video beam was visible on the viewer card when OMC flashes and it was hitting the steering mirror (but we need a viewport simulator to see if the beam will clear the viewport aperture).

Bringing the beam up higher on OM2

Unfortunately the beam was still very low (~1/4"), however I was able to use OM1 alignment slider to bring the beam up on OM2 and use OM2/3 alignment sliders to still center the OMC QPDs. After this was done, OM2 PIT offset became large but OM1/OM3 offsets became low-ish. This was a very good sign as it's infinitely easier to mechanically tilt OM2 than OM1/OM3 due to superior mechanical design.

Anyway, by doing this, the beam height on OM2 went up by about 1/8" (see Rahul's pictures). It's still too low by 1/8", but bringing the beam up more would mean that OM3 DAC output will become large w/o mechanically relieving, which I didn't want to do, so I decided to stay here.

Mechanically relieving the OM2 PIT offset

Julian set the OM2 PIT slider gain to 0.75 (from 1), Rahul turned the balance mass screw on the upper mass of OM2 to compensate. We repeated the same thing four times (slider gain 0.75->0.5->0.25->0, each step followed by Rahul's mechanical adjustment). We had to adjust OM2 Y slider in the process to bring the beam back to the center of the OMC QPDs, but overall, this was a really easy process (did I mention that tip-tilt adjustment is not an easy thing to do?).

We ended up with this (we haven't measured the beam height again as OM2 was the only thing that moved, so the height numbers are from the previous table just for convenience).

I declared that this is a good place to stay. Rahul fixed the balance mass on OM2 upper mass.

Rahul also fixed the balance mass on OMCS.

Fast shutter path, WFS path, ASAIR path, OMCR path

We closed the fast shutter and the reflected beam goes to the high power beam dump.

We opened the fast shutter and checked the WFS path. The beam was already hitting one quadrant of WFSB but was entirely missing the WSFA. The beam was a bit low on the lens on the WFS sled, so I used two fixed 1" steering mirrors upstream of the WFS sled to move the beam up on the lens and keep the path reasonably level. See Rahul's pictures for the beam height. After this, both WFSs saw the light, and at this point we used pico to center both. We weren't able to see the beam reflected by the WFS but assume that it still hit the black glass.

We tried to see the ASAIR beam but couldn't. Since the beam is hitting the center of ASC-AS_C, we assume that the ASAIR beam will still hit the black glass.

OMCR beam was already hitting the OMCR photodiode, but the beam was REALLY close to the beam dump that's supposed to catch ghost beam. We temporarily moved the dump so the beam is about 5mm from the edge of the glass, but this might be too far. I'll find how close it's supposed to be from the past alog.

Couldn't check if PZT1 is working.

We tried to see if OMC length error signal makes sense when scanning the OMC length, but whenever the OMC is close to resonance there was a huge transient in the DCPD SUM as well as the length signal, probably the intensity noise (due to acoustics or jitter or something) is too much. We'll measure the capacitance of the PZT from outside.

Current status of LVEA

Laser hazard, HAM5 GV is closed, HAM6 and HAM7 curtains are closed.

Remaining tasks

• Laser is required: Move the irises on HAM6 to the current beam position.
• Laser is required: OMCR path (beam dump position).
• Laser is required: OMC trans video path. Use viewport simulator.
• Check OSEMs of OMCS and OMs, recenter if needed.
• Restore the shroud panels.
• Restore the OMC trans beamdump on the -Y side.
• PZT1 capacitance check, grounding check etc.

Vicky, Sheila, Camilla

While in HAM6, Sheila noticed they;d lost their SQZ beam. Vicky found that the SQZ_OPO_LR guardian had unlocked but not realized it as the variable lockloss threshold was too high. When Vicky relocked it, the beam came back fine. Plot attached showing H1:SQZ-SHUTTER_I_TRIGGER_OUT_DQ increasing but not getting over the incorrect lockoss threshold of 2800.

We edited SQZ_OPO_LR  LOCKING_SEED_DITHER state to wait 15 seconds after the boosts are turned on for the OPO to be fully locked before moving to  LOCKED_SEED_DITHER and calculating the lockloss threshold (p.round(cdu.avg(1,'H1:SQZ-SHUTTER_I_TRIGGER_OUT_DQ')*1.025)), rather than taking the average while the seed is still finishing locking. If the seed dither unlocks, SQZ_MANGER will relock it.

Strangely the HAM6 QPDs (e.g. OMC_A) NSUM went negative (rather than 0) when this happened, making Tony and the CDS team check there was no issues. Unsure why it did.

Dana, Louis

We analyzed every measurement of the sensing function taken between the start of O4 and October 27th to see if they were reliable and came up with the following, summarized in the table below:

Ideally, the detector should be in lock state at least three hours before making a sensing function measurement to make sure the thermalization process is complete. However, there were a couple measurements that were made when the detector had only been locked for about two hours (06/21, 08/02), and there was one particularly problematic measurement that was made when the detector had only been locked for about 10 minutes (05/05). This last measurement should certainly not be included in the GPR calculation.

The code used to obtain the detector lock state and history given a report ID is attached below. Note: To run this code, you will need access to pydarm, so run the following command in the terminal before executing the file: source /ligo/groups/cal/local/bin/activate

[Julian, Naoki, Camilla, Sheila, Vicky]

Summary to get SQZ alignment beam: Launched 76mW into seed fiber, ~25 mW incident on opo cavity, ~0.85 mW transmitted through opo cavity. Had to find opo transmission past the VIP, for this we used green SK path as a reference. This ~0.85 mW opo transmission was bright on an IR card at the HAM5 gate valve, and enough to iris the SQZ beam in HAM7 and HAM6 (for OMC work 75512). DC 3/4 centering loops engaged easily, then OMC A/B QPD's saw the sqz beam.

----------- Notes from today ------------------------------------------

Launched power into SEED fiber (SQZT0): 76 mW

OPO IR REFL (CLF_TRIG_REFL_DC_POWERMON @ SQZT7): 24.8mW   (when opo is dither-locked).

Fiber rejected power PD in HAM7 (CLF_REFL_REJ) is 5.3 mW.
  -->  Seed fiber coupling: ~34% of the seed fiber launched power was incident on the opo cavity. 
  -->  40% coupling through fiber, ~6% mispolarized and rejected after fiber. This is similar to recent fiber alignment 75344, even after more recent on-table work 75486.

We had to find the OPO IR transmitted beam after the VIP. Nothing at first despite restoring suspensions 75502. Notes to self on what worked to find the SQZ beam post-vent:

• Started with no opo trans after the VIP, despite checking SQZT7 periscopes and the HAM7 RLF QPD's (which we use for FC WFS).
• 75502 saw ZMs had shifted after vent. Naoki 1) offloaded all FC+SQZ ASC and set all PSAMS offsets back to "nominal", then 2) he and Austin restored ZM1-6 + FC1 slider values to the same P/Y damp values as we last had in observe. Still did not see beam past the VIP (nothing on RLF QPD's, nothing on SQZT7 IR camera).
• SK path was helpful to find the VIP transmitted beam.
• Looking at the bottom SK periscope mirror on SQZT7 like here, we walked FC1 P/Y sliders to overlap FC_GR_REFL (which sees FC REFL), with OPO_REFL (which does not see FC REFL, and thus is a good alignment reference for the FC REFL path including ZM1-2-3-FC1).
• Used FC1 P/Y sliders to overlap FC_GR_REFL with OPO_REFL (looking at an IR card on the periscope lower mirror). While doing this, we found OPO IR TRANS on the RLF QPD's.
• Then we walked FC1 and ZM3 to maximize transmitted power, ie OPO_IR_DC_POWERMON. Not totally optimized, but seemed reasonable to keep going.

OPO IR TRANS (OPO_IR_PD_DC_POWERMON @ SQZT7): 0.85 mW  -- Just before opening the HAM7/5 gate valves. Opened the beam diverter, SQZ beam was bright on an IR card held at the HAM5 gate valve.

After opening gate valve, immediately saw the beam on AS A/B/C QPDs.

ASC-AS_A/B_DC_SUM_OUTPUT ~ 60.  ASC-AS_C_NSUM_OUT16 ~ 0.65-0.67.  

HAM6 crew irising SQZ beam, 75512.

HAM7 crew irising SQZ beam. Julian has some photos of HAM6 and HAM7 irses. Sheila -- looks like there is some clipping on the VIP (we have not totally optimized FC_REFL path slider alignments post-vent, just found the beam). Revisit this FC REFL alignment later.

Engaged DC 3/4 centering. It just worked. Control signals near 0. 

We see the beam on the OMC QPD's, power is consistent with ASC-AS_C power, around 300-400e-6 on each OMC QPD A/B. Power goes away when SQZ beam diverter is closed. See omc powers screenshot with as_wfs powers.

TO-DO SQZ work later:

• more photos of HAM7 irises
• measure sqz beam powers with power meter
• undo VIP clipping (using e.g. FC1/ZM3) - we didn't fully optimize this FC REFL alignment yet after venting.

I made a comparison of DARM_IN1_DQ and CAL-DELTAL_EXTERNAL_DQ in nominal VS new DARM offloading scheme (the new scheme itself is explained in alog 74887). Data for the NEW_DARM configuration was taken from Dec/21 (alog 74977) when Louis and Jenne successfully transitioned but with calibration that did not make sense.

The main things you must look at are the bottom left panel red and blue, i.e. the coherence between DARM_IN1 and CAL-DELTAL_EXTERNAL in the NEW (red) VS the old (blue) configuration. Blue trace is almost 1 as it should be, but the red drops sharply between 20Hz and 200Hz.

This does not make any sense because CAL-DELTAL_EXTERNAL is ultimately a linear combination of DARM_IN1 and DARM_OUT (see https://dcc.ligo.org/G1501518). Since DARM_OUT is linear to DARM_IN1, no matter where and how the noise is generated and no matter how you redistribute the signal in the ETM chain, CAL_DELTAL_EXTERNAL should always be linear to DARM_IN1, therefore coherence should be almost 1.

So what's the issue here?

The only straightforward possibility I see is that somehow excessive numerical noise is generated in the calibration model even with the frontend's double precision math. Maybe something is agressively low-passed and then high-passed, or vice versa, that kind of thing.

It is not an artefact of the single precision math of DTT. Both CAL_DELTAL_EXTERNAL and DARM_IN1 is already well whitened, and they're entirely within the dynamic range of single precision. For example, RMS of red CAL-DELTAL_EXTERNAL_DQ trace is ~7E-5 cts. From that number, I'd expect that the noise floor due to single precision is very roughly O(7E-5/10**7 /sqrt(8kHz)) ~ O(1E-13) cts/sqrtHz if it's close to white, give or take some depending on details, but the actual noise floor is ~10E-8 cts/sqrtHz. Same thing can be said for DARM_IN1.

It's not the numerical noise in DARM filter as the coherence between DARM_IN and SUS-ETMX_L3_LSCINF_L_IN1 (which is the same thing as DARM_OUT for coherence purpose) is 1 from 1Hz to 1kHz for both configurations (old -> brown, new -> green).  (It looks as if the coherence goes down above 1kHz for the old config, but that's irrelevant for this discussion, and anyway it's an artefact of DTT's single precision math.  See e.g. the top left blue (old config DARM_OUT) with RMS of 20k counts, corresponding to O(2E-5)/sqrtHz single noise floor due to single precision, give or take some. See where the actual noise floor is.)

It's not a glitch, noise level of CAL_DELTAL_EXTERNAL spectrum didn't change much from one fft to the other for the entire window (I used N=1 exponential to confirm this).

Note that there's also a possibility that excessive noise is generated in the SUS frontend too, polluting DARM_IN1 for real, not just for calibration model. I cannot tell if that's the case or not for now. The difference between the green (new) and brown (old) DARM_IN1 spectrum in the top left panel could just be a difference in gain peaking due to different DARM loop shape.

I'll see if double precision channels (recorded as double) in calibration model are useful to pinpoint the issue. Erik modified the test version of DTT so it handles the double precision numbers correctly without casting into single, but it's crashing on me at the moment.

Louis, Jenne, TJ, Sheila

Today we continued to try to transition to the new Darm configuration, which we had suceeded in doing in December but weren't able to repeat last week (75204). 

In our first attempt today we tried a faster ramp time, 0.1 seconds.  This caused immediate saturation of ETMX ESD.  We struggled to relock because of the environment. 

Because Elenna pointed out that the problem at roughly 3 Hz with the earlier transition attempts might have been the soft loops, we thought of trying to do the transition before the soft loops are engaged, after the other ASC is on.  We tried this first before transitioning to DC readout which wouldn't work because of the DARM filter changes.  Then we made a second attempt at DC readout.  We also lost lock due to a 2 Hz oscialltion, even without the soft loops on. 

Some gps times of transitions and attempt:

• 1387034100  succsesful transition from December
• 1388520329 attempt from last week with 5 second ramp
• 1388520329 attempt from last week to transition to the NEW DARM in TRANSITION_FROM_ETMX and LOWNOISE_ESD_ETMX with a 5 second ramp
• 1388952732 attempt from today to transition with a 0.1 second ramp, imediate saturation of ESD
• 1388961745 attempt at 2W with soft loops off (5 second ramp)