TITLE: 12/12 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 4mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.32 μm/s
QUICK SUMMARY:

• We've been locked for 10 hours, range looks lower than the start of the lock
• 2ndary microseism is trending up again
• Low wind
• Comissioning plans are from 9-12 PST, 17-20 UTC, calibration measurement at 16:30
• I power cycled the dust monitor in the vac prep lab but it still shows as disconnected

TITLE: 12/12 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: One lockloss this shift with an unknown cause, but otherwise a quiet evening with no sign of The Noise seen last night. H1 has now been observing for about 30 minutes.

• 02:31 - EQ mode activated for M5.6 quake from near Fiji
• 04:00 - Lockloss - alog81781
  • DRMI alignment looked poor so opted for initial alignment
• 05:31 - Observing

Lockloss @ 04:00 UTC - link to lockloss tool

No obvious cause, but maybe a very slight ETMX glitch right before the lockloss. Ends lock stretch at 26:51.

J. Oberling, R. Short

This afternoon, Jason and I started to look into why the FSS has been struggling to relock itself recently. In short, once the autolocker finds a RefCav resonance, it's been able to grab it, but loses it after about a second. This happens repeatedly, sometimes taking up to 45 minutes for the autolocker to finally grab and hold resonance on its own (which led me to do this manually twice yesterday). We first noticed the autolocker struggling when recovering the FSS after the most recent NPRO swap on November 22nd, which led Jason to manually lock it in that instance.

While looking at trends of when the autolocker both fails and is successful in locking the RefCav, we noticed that the fastmon channel looks the most different between the two cases. In a successful RefCav lock (attachment 1), the fastmon channel will start drifting away from zero as the PZT works to center on the resonance, but once the temperature loop turns on, the signal is brought back and eventually settles back around zero. In unsuccessful RefCav lock attempts (attachments 2 and 3), the fastmon channel will still drift away, but then lose resonance once the signal hits +/-13V (the limit of the PZT as set by the electronics within the TTFSS box) before the temploop is able to turn on. I also looked back to a successful FSS lock with the NPRO installed before this one (before the problems with the autolocker started, attachment 4), and the behavior looks much the same as with successful locks with the current NPRO.

It seems that with this NPRO, for some reason, the PZT is frequently running out of range when trying to center on the RefCav resonance before the temploop can turn on to help, but it sometimes gets lucky. Jason and I took some time familiarizing ourselves with the autolocker code (written in C and unchanged in over a decade) to give us a better idea of what it's doing. At this point, we're still not entirely sure what about this NPRO is causing the PZT to run out of range, but we do have some ideas of things to try during a maintenance window to make the FSS lock faster:

• Reduce delay turning on temploop (state 2; currently 1.0 sec)
• Increase wait time for mode to be centered (state 1; currently 0.1 sec; LLO uses 5.0 sec)
• Allow more voltage to PZT (requires using newer version of TTFSS box)
  • Current TTFSS limits PZT voltage to 13V-15V max
  • HV capable of supplying up to 65V
  • PZT capable of running up to 65V-100V

Starting from the somewhat strange RM1 spectra I saw earlier today (alog ), I have been looking at HAM1-related things.  I don't think this is a strong correlation, and maybe this just means that the HAM1 FF is doing what it should be, but it seems that the TTL4Cs on HAM1 are qualitatively different between times of good range and poor range.  However, the confusing thing is that the L4Cs seem bad at times when the range is good, which means that I don't really understand how they could be causing our troubles.  Also, other times seem to not have this inverse pseudo-correlation.  So, I'm not so sure that this is a sign of our troubles, or just something totally unrelated.

Sheila found that H1:TCS-ITMX_CO2_ISS_CTRL2_OUT_DQ stepped up from -0.2 to 0 on 2024/12/05 21:40:07 UTC (13:40 PST) Plot attached. This is of interest as this channel has ben a witness to our noisy/low range periods in DARM but is not connected to anything. I see no reason for this step up, the only person in the LVEA at the time was Robert setting up VP measurements (near HAM3 not CO2X) 81628, the CO2 laser remained l locked and we were not touching the CO2 chiller around the time 81634.

Since this step up, this channel has not been a good witness of our DARM noise, maybe the cable wasn't grounded and something changed to ground it on 12/05. Plot of it being a witness to the noise on 12/02 and not on 12/11.

Before and after the step up, this CO2 channel is still a witness to the CO2 rotation stage moving, attached. Both CO2X and CO2Y ISS CRTL2 channels see the rotation stages move, some crosstalk in chassis? CO2Y signal is orders of magnitude larger.  Jason was looking into these channels and the chassis

TITLE: 12/12 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: We stayed locked the entire shift, over 23 hours as of 00:30 UTC. Lost of investigations into the range drops today, range has been steady since the last occurance ~14hours ago. If the drop happens again there's a plan to investigate out of observing alog81774.
LOG:                                                                                                                                                               

• The VAC prep dust monitor is offline, I also reset the DM alarm levels that were reset yesterday following E1600132
• 18:24 UTC Tyler and the crane crew started down the gravel road to the FCES, into the enclosure at 18:26 and they left at 18:35 UTC (tagging DetChar)
• ~22:00 UTC I noticed the SQZ_PMC GRD notifying about the PMC VOLTs being low
  • 22:05 UTC the SQZer lost lock and dropped us out of Observing
  • The FC struggled to relock, FC2 P was very misaligned. I brought it closer to where it was when we last locked until it was close enough for FC-ASC to take over
  • 22:22 UTC Observing
• After lots of trending I think I found another ALS channel thats witnessing something related to the range drops, H1:ALS-Y_WFS_B_DC_YAW_OUT16 but I imagine its related to the 2 ALS_Y_WFS_B_DC channels IDed in alog81764
• 23:40 UTC BSC3 vac pressure glitch at PT132

I can't find the right alog for directions, so here is an easily searchable set of directions for HAM1 FF.

There is a master switch for the HAM1 feedforward, but it can sometimes cause problems to slam it on and off that way. Instead, you can ramp the input to the feedforward down to zero. From sitemap:

SEI > ISI Sensor Config > [middle of the screen, see attachment] HAM1 ASC FF > L4CINF

This opens a filter bank page with four filter banks. They each have a gain of 1 and a ramp time of 20 seconds. Set all of these gains to zero to turn off the input to the feedforward. Ramp them back to 1 to engage.

As several of us just talked about in the control room, if The Noise is happening when folks are on site and there's a plan of a thing to try, please feel free to drop Observing to check.  I think we'll keep the list of things to try elsewhere more dynamic than the alog (probably the LHO commissioning google doc).  Some examples of things that we're thinking of right now are (a) turning off the HAM1 FF, or (b) walking (gently) around electronics racks (CER, EX ESD driver area) to see if we can hear any electronics 'whining' or otherwise going bad.

Please do send me a mattermost message, which should audibly ping my phone, but this is causing such problems for our data quality that there is no need to wait for a response from me before trying something.

TITLE: 12/12 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY: H1 has been observing for 23 hours. The vac prep lab dust monitor is reporting as disconnected.

This alog presents the first steps I am taking into answering the question: "what is the calibrated residual test mass motion from the ASC?"

As a reminder, the arm alignment control is run in the common/differential, hard/soft basis, so we have eight total loops governing the angular motion of the test masses: pitch and yaw for differential hard/soft and common hard/soft. These degrees of freedom are diagonalized in actuation via the ASC drive matrix. The signals from each of these ASC degrees of freedom are then sent to each of the four test masses, where the signal is fed "up" from the input of the TST ISC filter banks through the PUM/UIM/TOP locking filters banks (I annotated this screenshot of the ITM suspension medm for visualization). No pitch or yaw actuation is sent to the TST or UIM stages at Hanford. The ASC drive to the pum is filtered through some notches/bandstops for various suspension modes. The ASC drive to the TOP acquires all of these notches and bandstops and an additional integrator and low pass filter, meaning that the top mass actuates in angle at very low frequency only (sub 0.5 Hz).

Taking this all into account involves a lot of work, so to just get something off the ground, I am only thinking about ASC drive to the PUM in this post. With a little more time, I can incorporate the drive to the top mass stage as well. Thinking only about the PUM makes this a "simple" problem:

• we know exactly what the ASC drive is in counts- we just measure the "H1:ASC-[DOF]_[P/Y]_OUT_DQ" channel
• We propagate drive that to each test mass knowing the drive matrix we have set
  • screenshots of the pitch matrix and yaw matrix
• We calibrate this from drive counts to Newton*meters knowing the drive calibration from G1100968
  • for PUM: 40 V /2**20 ct [DAC] * 0.268 mA / V [drive strength] * 0.0309 N / A [force coeff] * 70.7 mm [lever arm], pit/yaw lever arm is the same for the PUM
  • I conveniently chose a time from before the ETMX DAC change to avoid thinking too hard about the DAC counts
• We transform this N*m PUM drive to radians of test mass motion using the PUM-TST state space suspension model transfer function, which is calibrated into real units
  • I did all of this in python, so I used Kevin's exported quad state space model, found here: T2300299
  • I used the damped model which applies the ETMX top mass damping filters

I have done just this to achieve the four plots I have attached to this alog. These plots show the ITM and ETM test mass motion in rad/rtHz from each degree of freedom and the overall radian RMS value. That is, each trace is showing you exactly how much radians of motion each ASC degree of freedom is sending to the test mass through the PUM drive. The drive matrix value is the same in magnitude for each ITM and each ETM, meaning that the "ITM" plot is true for both ITMX and ITMY (the drives might differ by an overall sign though).

Since I am just looking at the PUM, I also didn't include the drive notches. Once I add in the top mass drive, I will make sure I capture the various drive filters properly.

Some commentary: These plots make it very evident how different the drive is from each ASC degree of freedom. This is confusing because in principle we know that the "HARD" and "SOFT" plants are the same for common and differential, and could use the same control design. However, we know that the sensor noise at the REFL WFS, which controls CHARD, is different than the sensor noise at the AS WFS that control DHARD, so even with exact same controller, we would see different overall drives. We also know that we don't use the same control design for each DOF, due to the sensor noise limitations and also the randomness of commissioning that has us updating each ASC controller at different times for different reasons. For example, the soft loops both run on the TMS QPDs, but still have different drive levels.

Some action items: besides continuing the process of getting all the drives from all stages properly calibrated, we can start thinking again about our ASC design and how to improve it. I think two standout items are the SOFT P and CHARD Y noise above 10 Hz on these plots. Also, the fact that the overall RMS from each loop varies is something that warrants more investigation. I think this is probably related to the differing control designs, sensor noise, and noise from things like HAM1 motion or PR3 bosems. So, one thing I can do is project the PR3 damping noise that we think dominates the REFL WFS RMS into test mass motion.

As part of a different investigation (still trying to understand why our range goes bad sometimes), I incidentally may have found a source for some glitches / range drops. 

I'm not going to look further into this, but instead tag detchar-request in hopes that someone else has some time to think about it.  I'll also directly send messages to Jim and Elenna, who manage this feedforward system.

In the attached screenshot there are a few channels that include 'TTL4C' - those are the tabletop L4C seismic sensors on HAM1.  There are also 'FFHAM1' channels that take a channel derived from those L4Cs, and feed it forward to the error signal of the ASC loop that is referred to in the name.  A moment or so after there is a glitch in those channels, there is a range drop in DARM. 

I will say that when I zoom in, it looks like the glitch appears in the FFHAM1 channel before it appears in the TTL4C channel, but based on my understanding of the signal flow, I'm not entirely sure how that's possible.  I'm hoping that Elenna (who knows much more than I do) can help think this through.

TITLE: 12/11 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.32 μm/s
QUICK SUMMARY:

• We've been locked for 14:20, in observing since 03:33
• 2ndary microseism is decreasing, its just below the 90th percentile
• Low wind

As part of WP 12239 we moved h0epics, cdsvmscript1, epics-burt, autoburt off of the cds0proxmox hypervisor.  This resulted in a few minutes of down time for the dust monitor IOC around 8:19am localtime.

After this we done we were able to look at cdsproxmox.  Its boot drive had failed.  After some help from Fil we got the drives replaced and reinstalled proxmox ve on cdsproxmox.  We adjusted DNS and renamed the box cdsproxmox0.

Some notes:

 * This is now in a temporary state.  We aim to retire this hardware by or at the end of O4.  New hypervisor computers are being procured.  As such we did not provision much storage on this.  Just enough to run the hypervisor, relying on the shared storage layer to handle the VM.

 * As per the proxmox administrators guild we removed cdsproxmox from the cluster prior to attaching it back as cdsproxmox0.