This work is following up on LHO:73735. 

I've written a script that retroactively fixes the pyDARM parameter issues discussed in LHO:73735. The script lives here on the CDS workstations: /ligo/home/louis.dartez/projects/20240124_script_fix_bad_inis_from_alog_LHO73735/fix_bad_site_inis.py.

Since the pyDARM parameter INI files were initially written in error, anyone trying to process old corresponding to the affected reports' times would be using the wrong IFO models. As such, I've used the script above to fix the affected reports in situ. The original reports have been copied into /ligo/groups/cal/H1/reports/archive/reports_preserved_from_fix_for_LHO73735.

WP11646 New h1sqz model

Daniel, Dave:

A new h1sqz model was installed. DAQ restart was required

WP11651 Add New SQZ_PMC Guardian nodel to DAQ

Vicky, Camilla, Daniel, Dave:

The new SQZ_PMC GRD node was added to H1EPICS_GRD.ini. DAQ + EDC restart required

DAQ Restart

Dave:

The DAQ was restarted for the above changes. Sequence was 0-leg, EDC, 1-leg.

No major problems with the restart, both GDS daqds had to be restarted a second time for channel list synchronization.

TITLE: 01/30 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: Tony
INCOMING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 5mph 5min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.46 μm/s
QUICK SUMMARY:

Revovering from the Dolphin timing crash desribed in Alog 75612
We have taken all the SUS pensions to safe as soon as I could.
Dave restarted all the IOPs.
Rahul Recovered all the SUS from Safe
The CDS issues seem to be resolved by 17:28 UTC

Dainel started working on Beckhoff restarts, Ran into issues at EY.

SQZ Model was restarted which did effect FW0 & FW1 so there is a breif time of no data being written to the frames.

Moving Forklift in the LVEA

18:00 UTC Beckhoff slow controls just went down across the Site including the PSL.
18:04 UTC Beckoff back online.

 
PSL_ENV_LASERRM_ACN_TEMP_DEGF alarm 18:33 UTC
Trended the channel, looks fine.

Beam Splitter ISI tripped.
Beam Splitter ISI Reset.
Beam Splitter SUS is in Damped.

The LVEA is Now LASER SAFE for the HAM3 Door removal

day 1 (alog 75548), day 2 (alog 75557), day 3 (alog 75575), day 4 (alog 75601)

Recovery from reboots

After some things were rebooted, we've found that the suspension slider offsets for all OMs, SRM and ZM5 were reverted back to old-ish numbers. I and Camilla manually restored that.

Camilla saw that the beam was not quite back to the old position on one of the HAM7 QPDs but wasn't that bad.

The beam was already on ASC-AS_C but not centered, so I centered it using SRM.

Following that, I had to make a minor tweaking of OM1/2/3 to recenter OMC qpds quickly.

HAM6 irises are centered

Preet and Sheila recentered the two irises on HAM6. From this point on, these irises are a fiducial for the IFO beam.

OMC trans video beam and OMCR beam dump will be done later this week

LVEA was transitioned to laser safe for HAM3 door removal. We'll continue suspension work in HAM6.

Jonathan, Erik, Tony, Rahul, Richard, Dave:

Last night we had another one of the corner station glitches which took down all the models on h1susb123, h1sush2a, h1sush34 and h1sush56.

The glitch happened at 18:45:55 PST Mon 29th Jan 2024 during the O4a,b break.

This has happened several times before, details in FRS30324, with a regularity of every 80 to 100 days.

After the glitch, the dolphin IPC continued to function, which meant that the SWWDs did not trip the associated Seismic systems.

DMESG logs did not show anything.

I ran a complete set of Dolphin DIS_DIAGS (previous run was 24th Jan 2024), which only showed the issues with OAF0 and SEIH16 from last Friday's crashes. No problems with any SUS was seen.

Recovery from the crash was:

• Set all SEI IOP SWWD system into BYPASS mode with a very long bypass time (999999999S)
• Restart all the models on h1sus[b123, h2a, h34, h56]. All started with no problems.
• Unbypass the SEI SWWDs.
• Hand over to control room for SUS recovery.

The H1SQZ model has been updated to fix the PMC_TRANS & FIBR_PD readbacks, and to add fast DQ channels for the new PMC readbacks.

The EtherCAT system was been updated to fix the PMC_REFL PD type.

A new squeezer guardian node has been added for the PMC which required a DAQ restart to add the new channels.

Tue Jan 30 10:12:06 2024 INFO: Fill completed in 12min 2secs

Jordan confirmed a good fill curbside, he removed some ice buildup around the discharge line vent.

TITLE: 01/30 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 4mph Gusts, 3mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:
The is a fairly large number of red items on the CDS overview, which is likely due to Timing Glitch on SUS Front End ALOG 75612

There was a timing glitch that affected many models running on corner station front-ends, and the IOPs on h1susb123, h1sush34, h1sush2a, h1sush56.

The timing error is on the order of tens of milliseconds, which is consistent with a Dolphin glitch.  Non-Dolphin front-ends and end station front ends were not affected.

Error was at about Jan 30, 02:45 UTC.  Models are running, but the affected IOPs will need to be restarted.

Workstations were updated and rebooted.  This was an OS package update.  Conda packages were not updated.

day 1 (alog 75548), day 2 (alog 75557), day 3 (alog 75575)

HAM7 irises were good

Sheila/Camilla checked the iris position on HAM7 and it was good.

ASC-AS_C whitening gain was increased by 18dB, dark offset was reset

I didn't like that the ASC-AS_C input was so small. Increased the whitening gain by 18dB (from nominal 18dB to 36dB) and reset the dark offset.

Recentered the beam on ASC-AS_C. One thing that was strange was that the ASC-AS_C_NSUM would become MUCH bigger (like a factor of 10) when SRM is misaligned. I was worried that I was looking at a ghost beam. Camilla measured the  beam power to be ~1mW out of HAM7 and ~0.7mW into HAM6. When ASC-AS_C was centered, ASC-AS_C_NSUM_OUT became ~0.008 give or take some. Taking the 16dB extra whitening (i.e. a factor of 8) into account, ASC-AS_C_NSUM~0.008 means about 1mW into HAM6, which was in a ballpark, so I convinced myself that the beam on AS_C was good.

HAM6 irises and beam height on OM, the beam was still very low on OM2

At this point OMC QPDs are reasonably centered, so Sheila and Camilla checked the beam position on irises in HAM6.

The beam was OK on the first iris but was a bit low (~2mm) on the iris closer to the OM1.

The beam position on OMs at this point as well as the slider values and max DAC output are listed below (see Camilla's pictures, too). Note that the YAW position in the table is the position of the incoming beam on the mirror measured at some (unknown) distance, it's not on the mirror.  

The beam was clearing the input and output hole on the shroud, was cleanly hitting the small OMCR steering mirror by the cage, and was already going to the OMCR diode.

They confirmed that the OMC trans video beam was visible on the viewer card when OMC flashes and it was hitting the steering mirror (but we need a viewport simulator to see if the beam will clear the viewport aperture).

Bringing the beam up higher on OM2

Unfortunately the beam was still very low (~1/4"), however I was able to use OM1 alignment slider to bring the beam up on OM2 and use OM2/3 alignment sliders to still center the OMC QPDs. After this was done, OM2 PIT offset became large but OM1/OM3 offsets became low-ish. This was a very good sign as it's infinitely easier to mechanically tilt OM2 than OM1/OM3 due to superior mechanical design.

Anyway, by doing this, the beam height on OM2 went up by about 1/8" (see Rahul's pictures). It's still too low by 1/8", but bringing the beam up more would mean that OM3 DAC output will become large w/o mechanically relieving, which I didn't want to do, so I decided to stay here.

Mechanically relieving the OM2 PIT offset

Julian set the OM2 PIT slider gain to 0.75 (from 1), Rahul turned the balance mass screw on the upper mass of OM2 to compensate. We repeated the same thing four times (slider gain 0.75->0.5->0.25->0, each step followed by Rahul's mechanical adjustment). We had to adjust OM2 Y slider in the process to bring the beam back to the center of the OMC QPDs, but overall, this was a really easy process (did I mention that tip-tilt adjustment is not an easy thing to do?).

We ended up with this (we haven't measured the beam height again as OM2 was the only thing that moved, so the height numbers are from the previous table just for convenience).

I declared that this is a good place to stay. Rahul fixed the balance mass on OM2 upper mass.

Rahul also fixed the balance mass on OMCS.

Fast shutter path, WFS path, ASAIR path, OMCR path

We closed the fast shutter and the reflected beam goes to the high power beam dump.

We opened the fast shutter and checked the WFS path. The beam was already hitting one quadrant of WFSB but was entirely missing the WSFA. The beam was a bit low on the lens on the WFS sled, so I used two fixed 1" steering mirrors upstream of the WFS sled to move the beam up on the lens and keep the path reasonably level. See Rahul's pictures for the beam height. After this, both WFSs saw the light, and at this point we used pico to center both. We weren't able to see the beam reflected by the WFS but assume that it still hit the black glass.

We tried to see the ASAIR beam but couldn't. Since the beam is hitting the center of ASC-AS_C, we assume that the ASAIR beam will still hit the black glass.

OMCR beam was already hitting the OMCR photodiode, but the beam was REALLY close to the beam dump that's supposed to catch ghost beam. We temporarily moved the dump so the beam is about 5mm from the edge of the glass, but this might be too far. I'll find how close it's supposed to be from the past alog.

Couldn't check if PZT1 is working.

We tried to see if OMC length error signal makes sense when scanning the OMC length, but whenever the OMC is close to resonance there was a huge transient in the DCPD SUM as well as the length signal, probably the intensity noise (due to acoustics or jitter or something) is too much. We'll measure the capacitance of the PZT from outside.

Current status of LVEA

Laser hazard, HAM5 GV is closed, HAM6 and HAM7 curtains are closed.

Remaining tasks

• Laser is required: Move the irises on HAM6 to the current beam position.
• Laser is required: OMCR path (beam dump position).
• Laser is required: OMC trans video path. Use viewport simulator.
• Check OSEMs of OMCS and OMs, recenter if needed.
• Restore the shroud panels.
• Restore the OMC trans beamdump on the -Y side.
• PZT1 capacitance check, grounding check etc.

Today's activities:
- All bolts (but 4 of them) on HAM3 Y+ door have been broken, in preparation for the door removal - most likely tomorrow, during laser safe conditions
- The cable trays on BSC8 have been removed, as they would be on the way for the 12" and 16.5" conflat feedthru removals - this activity will start after the LVEA will be laser safe
- The troubleshooting of the EX RGA ion pump has been done - see here in detail: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=75605
- The corner purge air was measured, and found satisfactory (~-46 deg C); seed details in comment

Vicky, Sheila, Camilla

While in HAM6, Sheila noticed they;d lost their SQZ beam. Vicky found that the SQZ_OPO_LR guardian had unlocked but not realized it as the variable lockloss threshold was too high. When Vicky relocked it, the beam came back fine. Plot attached showing H1:SQZ-SHUTTER_I_TRIGGER_OUT_DQ increasing but not getting over the incorrect lockoss threshold of 2800.

We edited SQZ_OPO_LR  LOCKING_SEED_DITHER state to wait 15 seconds after the boosts are turned on for the OPO to be fully locked before moving to  LOCKED_SEED_DITHER and calculating the lockloss threshold (p.round(cdu.avg(1,'H1:SQZ-SHUTTER_I_TRIGGER_OUT_DQ')*1.025)), rather than taking the average while the seed is still finishing locking. If the seed dither unlocks, SQZ_MANGER will relock it.

Strangely the HAM6 QPDs (e.g. OMC_A) NSUM went negative (rather than 0) when this happened, making Tony and the CDS team check there was no issues. Unsure why it did.