Tue Dec 10 10:10:41 2024 INFO: Fill completed in 10min 38secs

Camilla, Robert, WP12232

While looking for coupling of CO2 chillers into DARM at Crab pulsar 59Hz 81246, and also CO2s effect on SQZ 72244, this morning we turned off the CO2s and stayed locked ~40 minutes which is longer than we expected. Unsure of the lock loss cause as maintenance activities had started.

We aimed to see if the 59Hz coupling to DARM peak changed (needs ~20 minutes+ of data), but the accelerators showed that by increasing the CO2 power dumped in the water cooled beam dump, the load on the chillers changed and the 59Hz peak moved ~0.1Hz which means that it would be very difficult to see in DARM.

• 15:41 Turned CO2X and CO2Y power output down to 0W.
• 15:51 closed iris at bottom of CO2Y periscope (there is no place in VP for guillotine)
• 16:01 onwards, Robert was doing backscatter tests on HAM3, making IFO glitches/noise
• 1624 Opened CO2Y irirs, IFO lost lost lock

In attached plot, can see that the SQZ (especially at high frequency) got worse without CO2s. The SQZ ASC also changed the alignment mainly in YAW, does that mean that the CO2s need some alignment improvements in YAW? Looking at HWS signals, the optics substrate absorption changes as expected and started to level out, although the simulation didn't expect them too level out yet.

In future, to repeat this test, we could dump to 1.7W going into DARM using a normal beam-dump to dump the 1.7W before the periscope so that the load on the chiller isn't changed but the CO2 is not on being injected and anything that could backscatter light after the periscope is blocked. Alternatively, we could shake the table. Still the VPs themselves could be a source of scatter as are only AR coated at 10.6um, meaning they reflect up to 15% of 1064nm per surface DCC D1100439.

TITLE: 12/10 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 6mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:

• We've been locked for 13:22, we dropped observing at 15:25 for the sus charge and from 15:01 to 15:20 for the magnetic
• The magentic injections have run and the SUS charge are currently running

Workstations were upgraded and rebooted.  This was an OS packages upgraded.  Conda packages were not upgraded.

TITLE: 12/10 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Currently Observing at 158 Mpc and have been Locked for 4 hours. The range is okay now, but we did go through a patch of low range that looks very similar to the low range from the last lock. For both this lock and  the previous lock it looks like the low range started around the same time after locking, 1 hour and 20 minutes after the lock started(ndscope). I couldn't find anything with squeeze or the jitter diaggui.
LOG:

00:26 Running an initial alignment
01:26 Initial alignment done, relocking

02:06 NOMINAL_LOW_NOISE    
02:09 Observing  

Closes FAMIS#26020, last checked 81615

• HAM6 H2 and V2 are a bit higher between 50 and 60 Hz
• Beamsplitter stage 2 H1 is elevated between 1 and 4 Hz
• ETMY stage 1 all sensors are higher than normal between 3 and 6 Hz, with the V's noticably higher than they were last week.

TITLE: 12/10 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.37 μm/s
QUICK SUMMARY:

Late shift start log, but we have been relocking and are currently at REDUCE_RF45_MODULATION_DEPTH

Erik, Jonathan, and Tony,

As per WP 12239 we installed a new hypervisor machine, cdsproxmox2.  Erik and Tony racked a spare system in the server3 rack.  We configured the sw-msr-server3 switch to support the new hypervisor using ports 47 & 48.

We installed the same proxmox ve version that runs the current cluster via a bootable iso image.  The install went very smoothly.  We did a simple local install with zfs zraid and 3x2TB disks.  We were able to quickly add it to the cluster by using the cluster management page and the 'join cluster' command.

Some notes of extra steps we needed to do:

 * Add the proper proxmox apt-source (the no-subscription version)
 * Install the ifupdown2 package (after setting up the apt-source)
 * Then we could enable the 2nd ethernet port and create the VMs main ethernet bridge
   * eno2 was configured to autostart, no ip address
   * vmbr1 was configured  to autostart, no ip address, vlan aware, and it is associated with eno2

Then we could migrate some VMs.  We have moved ldap servers, a development machine, our conda mirror (in process) (these can be moved without disturbing the control room).

The point of this work is to provide enough physical hardware to be able to turn off one of the other hypervisors for maintenance work.  Cdsproxmox has disk issues.

This work will continue tomorrow.

TITLE: 12/10 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is LOCKING and in ENVIRONMENT due to EARTHQUAKE.

We had a few good hours of locking today but as with yesterday, earthquakes have been rampant. Here are details:

• Lockloss 1: EQ caused and interrupted comissioning.
  • Tripped ISI HAM7 WD after LL (during high motion)
  • alog 81694
• Lock Re-acquisition:
  • FSS glitchy during initial alignment, resulting in DOWN until FSS relocked and I was able to re-do the initial alignment. After that, locking was fully auto.
  • screenshot attached
• Lockloss 2: EQ caused
  • Tripped many WDs
  • alog 81709
• SQZ Recovery following interlock trip
  • HAM7 interlock tripped, turning off High Voltage and unlocking SQZ. Fil turned it on in the morning and we were able to lock SQZ again. Cause of trip yet unknown but VAC knows about it - there was no immediate culprit or pressure spike when the trip happened.
  • SQZ was successfully brought back using Ryan's alog 79887
  • relevant alogs: alog 81689, alog 81697
  • SDF diffs attached
• Range dips hour into observing
  • We've been seeing our range dip down by ~5-10MPc during observing and have been trying to troubleshoot. This is not a new occurrence.
  • Went into comissioning for 10 mins to turn SQZ off (22:36 - 22:46 UTC) to see the effect on range - we saw no positive range change but seemed to stabilize the noise of the plot.
  • Camilla and TJ turned off CO2 to see if this was a potential cause but they saw no positive range change. alog 81705
  • Investigation ongoing.
• FMCSSTAT Red on CDS Overview
  • Dave noticed FMCSTAT going red on CDS overview and found that the Zone2A LVEA temps were going up very quickly. He informed FAC. Alog 81701
  • Tyler confirmed that it was not an issue, just that the temperature differential today was high and that the Zone 2A thermometer is in the wall, making it more susceptible to such excursions.

LOG:

Lockloss @ 12/09 23:12UTC due to large earthquake from Nevada. Sitting in DOWN until ground motion goes down

Camilla C, TJ S

This morning we had another period where our range was fluctuating almost 40Mpc, previously seen on Dec 1 (alog81587) and further back in May (alog78089). Camilla and decided to turn off both TCS CO2s for a short period just to completely rule them out, since previously there was correlation with these range dips and a TCS ISS channel. We saw no positive change in DARM during this short duration test, but we didn't want to go too long and lose lock. CO2s were requested to have no output power from 16:12:30-16:14:30UTC

The past times that we have seen this range loss, the H1:TCS-ITMX_CO2_ISS_CTRL2_OUT_DQ channel and the H1:SQZ-FC_LSC_DOF2_OUT_DQ channel had noise that correlated to the loss, but theISS channel showed nothing different this time (attachment 2). We also were in a state of no squeezing at the time. So it's possible that this is a completely different type of range loss.

DetChar, could we run Lasso or check on HVETO for a period during the morning lock with our noisy range?

TITLE: 12/09 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Currently Observing without squeezing and have been Locked for 1 hour.
All thoughout the day we haven't been able to lock because of many earthquakes, but in the last hour I was finally able to relock the IFO without us losing lock from ground motion.
When we got to NLN we couldn't go into Observing because the squeezer PMC wouldn't lock - the message said "Cannot lock PMC. Check SQZ laser power". Last time this happened we were just able to get it to lock by re-requesting LOCKED for the PMC guardian, but this time it didn't work. I called Camilla, and we saw that although the PZT thought it was scanning to try and lock, the PZT volts were not changing and were just sitting out of range, around -12 volts, and that 10 hours ago something had happened to the PZT volts where they suddently glitched down to over -200, which is much lower than they ever go.
We weren't able to get it back working so we used Ryan Short's script to change the nominal sqz states. I accepted the sdfs, and we went to Observing at 06:31 UTC. Here is Camilla's alog with an ndscope. tagging SQZ
LOG:

22:32 Started an initial alignment
22:58 Initial alignment done, relocking
    - Lockloss from LOCKING_ARMS_GREEN
    - Lockloss from OFFLOAD_DRMI_ASC
    - Lockloss from ENGAGE_ASC_FOR_FULL_IFO
    - Lockloss from RESONANCE due to large EQ hitting
00:23 Put IFO in DOWN to wait out earthquake
- HAM2, HAM3, HAM5 ISI tripped
- IM2 tripped

02:58 Started an initial alignment
03:22 Initial alignment done, relocking
    - Lockloss from PREP_DC_READOUT_TRANSITION due to earthquake
03:43 Sitting in DOWN while earthquake passes

04:48 Trying to relock
05:46 NOMINAL_LOW_NOISE
    - Issues getting PMC to lock - see above
06:31 Observing without squeezing

It appears that the SQZ HV (PMC/SHG/OPO PZT, ZM2,4,5 PSAMS) went down at 12/08 20:33UTC. Trends attached. 

Oli found that the SQZ PMC wasn't locking. We tried enabling the PZT manually and although it thought it was scanning, the PZT Volts remained at -12, Oli found it had been like this since 12/08 20:33UTC. Then realized all PSAMS volts are off and other PZTs. Oli has taken the IFO into observing without SQZ for tonight do the CDS/EE team can look at this tomorrow.

Back in March (76269) Jeff and I had updated all the suspension watchdogs (besides OFIS, OPOS, and HXDS since those were already up to date) to use better blrms filtering and to be output into um. We set the suspension watchdog thresholds to values between 100 and 300 µm, but these values were set arbitrarily since there was no way to previously see how far the stages move during different scenarios. We had upped a few of the thresholds after having some suspensions trip when they probably shouldn't have, and this is a continuation of that.
During the large earthquake that hit us on December 5th, 2024 18:46 UTC, all ISI watchdogs tripped as well as some of the stages on several suspensions. After a cursory look, all suspensions that tripped only had either the bottom or bottom+penultimate stage trip, meaning that with the exception of the single suspensions, the others' M1 stage damping should have stayed on.

We wanted to go through and check whether the trips may have just been because of the movement from the ISIs tripping. If that is the case, we want to raise the suspension watchdog thresholds for those stages so that these suspensions don't trip every single time their ISI trips, especially if the amount that they are moving is still not very large.

Suspension stages that tripped:

Triples:
- MC3 M3
- PR3 M2, M3
- SRM M2, M3
- SR2 M2, M3

Singles:
- IM1 M1
- OFI M1
- TMSX M1

MC3 (M3) (ndscope1)

When the earthquake hit and we lost lock, all stages were moving due to the earthquake, but once HAM2 ISI tripped 5 seconds after the lockloss, the rate at which the OSEMs were moving quickly accelerated, so the excess motion looks to mainly be due to the ISI trip (ndscope2).

PR3 (M2, M3) (ndscope3)

Looks to be the same issue as with MC3.

SRM (M2, M3) (ndscope4)

Once again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM5 saturated and the ISI watchdogs tripped.

 SR2 (M2, M3) (ndscope5)

Again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM4 saturated and the ISI watchdogs tripped.

IM1 (M1) (ndscope6)

Again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM2 saturated and the ISI watchdogs tripped.

OFI (M1) (ndscope7)

Again, the OSEMs are moving after the lockloss due to the earthquake, but the rate they were moving at greatly increased once HAM5 saturated and the ISI watchdogs tripped.

TMSX (M1) (ndscope8)

This one seems a bit questionable - it looks like some of the OSEMs were already moving quite a bit before the ISI tripped, and there isn't as much of a clear place where they started moving more once the ISI had tripped(ndscope9). I will still be raising the suspension trip threshold for this one just because it doesn't need to be raised very much and is within a reasonable range.

Using the darm_integral_compare.py script from the NoiseBudget repo (NoiseBudget/aligoNB/production_code/H1/darm_integral_compare.py) as a starting point, I made a version that is simplified and easy to run for when our range is low and we want to compare range vs frequency with a previous time.

It takes two starting times, supplied by the user, and for each time, it grabs the DARM data between the start time and an end time of starttime+5400 seconds (1.5 hours). Using this data it calculates the inspiral range integrand and returns two plots(pdf) - one showing the range integrand plotted against frequency for each set of data(png1), and then the second plot just shows DARM for each set of data, along with a trace showing the cumulative difference in range between the two sets as a function of frequency(png2). These are saved both as pngs and in a pdf in the script's folder.

This script can be found at gitcommon/ops_tools/rangeComparison/range_compare.py. To run it you just need to supply the gps times for the two sets of time that you want to compare, although there is also an optional argument you can if you want the length of data taken to be different than the default 5400 seconds. The command used to generate the PDF and PNGs attached to this alog was as follows: python3 range_compare.py --span 5000

I've written a script, /opt/rtcds/userapps/release/sqz/h1/scripts/switch_nom_sqz_states.py, which will change the nominal states of the SQZ Guardian nodes to be used when it's decided that H1 will observe without SQZ, a process adapted from the ObservationWithOrWithoutSqueezing wiki. This is intended to save operators time so the switch to a configuration where H1 can observe without SQZ is quick and brings us to observing promptly.

The script first commits any uncommitted changes in the SQZ Guardians to svn with an "as found" message before the nominal states are changed. Once changed, the nodes are then loaded and SQZ_MANAGER is requested to NO_SQUEEZING, which should make sure all nodes are in the state they should be.

This script also adds the 'syscssqz' model to the EXCLUDE_LIST in the DIAG_SDF node when going to the no-SQZ configuration. This is to facilitate any SDF diffs that may appear as a result of SQZ misbehaving and will allow H1 to go to observing while ignoring any diffs in this model. More models to exclude can be added in the script as desired.

Since the pysvn package used by the script to commit to svn is a Debian package not found in conda, all conda environments must be deactivated for this script to work. Hence, when running this script, utilize the 'noconda' bash script wrapper found in userscripts. Calling this script to change to the configuration where H1 will observe without SQZ would look as follows:

noconda python switch_nom_sqz_states.py without

This script can also be used to go back to the configuration where H1 is observing with SQZ; simply replace the 'without' argument when running it with 'with' and the nominal SQZ Guardian states will be changed back to normal and the SQZ models will be removed from DIAG_SDF's exclude list.