Thu Dec 12 10:04:12 2024 INFO: Fill completed in 4min 9secs

Gerardo confirmed a good fill curbside. 

At 15:40:28 yesterday, Wed 11dec2024 we had another sensor glitch on the BSC vacuum gauge (PT132). This was another sharp 2 second wide spike in the pressure reading, which was detectec by VACSTAT as a delta-p trip.

VACSTAT went into SINGLE (sensitive) mode and the CDS ALARMS block on the overview went red because an alarm was sent to CDS.

This morning I cleared the alarm by resetting VACSTAT at 07:39 Thu 12dec2024 PST.

I ran the scheduled calibration measurements starting at 16:30 UTC following the wiki.

Broadband Start Time: 1418056220

Broadband End Time: 1418056588

Simulines Start Time: 1418056620

16:43:42 UTC EX saturation

Simulines End Time: 1418058008

Files Saved:

2024-12-12 17:00:28,850 | INFO | File written out to: /ligo/groups/cal/H1/measurements/DARMOLG_SS/DARMOLG_SS_20241212T163643Z.hdf5
2024-12-12 17:00:28,857 | INFO | File written out to: /ligo/groups/cal/H1/measurements/PCALY2DARM_SS/PCALY2DARM_SS_20241212T163643Z.hdf5
2024-12-12 17:00:28,862 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L1_SS/SUSETMX_L1_SS_20241212T163643Z.hdf5
2024-12-12 17:00:28,866 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L2_SS/SUSETMX_L2_SS_20241212T163643Z.hdf5
2024-12-12 17:00:28,870 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_20241212T163643Z.hdf5
ICE default IO error handler doing an exit(), pid = 900165, errno = 32
PST: 2024-12-12 09:00:28.919662 PST
UTC: 2024-12-12 17:00:28.919662 UTC
GPS: 1418058046.919662

Wed Dec 11 10:04:17 2024 INFO: Fill completed in 4min 14secs

Gerardo confirmed a good fill curbside. Late entry for yesterday's fill. Note Y marker on trend is displayed incorrectly as 70C, it is actually 65C.

TITLE: 12/12 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 4mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.32 μm/s
QUICK SUMMARY:

• We've been locked for 10 hours, range looks lower than the start of the lock
• 2ndary microseism is trending up again
• Low wind
• Comissioning plans are from 9-12 PST, 17-20 UTC, calibration measurement at 16:30
• I power cycled the dust monitor in the vac prep lab but it still shows as disconnected

TITLE: 12/12 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: One lockloss this shift with an unknown cause, but otherwise a quiet evening with no sign of The Noise seen last night. H1 has now been observing for about 30 minutes.

• 02:31 - EQ mode activated for M5.6 quake from near Fiji
• 04:00 - Lockloss - alog81781
  • DRMI alignment looked poor so opted for initial alignment
• 05:31 - Observing

Lockloss @ 04:00 UTC - link to lockloss tool

No obvious cause, but maybe a very slight ETMX glitch right before the lockloss. Ends lock stretch at 26:51.

J. Oberling, R. Short

This afternoon, Jason and I started to look into why the FSS has been struggling to relock itself recently. In short, once the autolocker finds a RefCav resonance, it's been able to grab it, but loses it after about a second. This happens repeatedly, sometimes taking up to 45 minutes for the autolocker to finally grab and hold resonance on its own (which led me to do this manually twice yesterday). We first noticed the autolocker struggling when recovering the FSS after the most recent NPRO swap on November 22nd, which led Jason to manually lock it in that instance.

While looking at trends of when the autolocker both fails and is successful in locking the RefCav, we noticed that the fastmon channel looks the most different between the two cases. In a successful RefCav lock (attachment 1), the fastmon channel will start drifting away from zero as the PZT works to center on the resonance, but once the temperature loop turns on, the signal is brought back and eventually settles back around zero. In unsuccessful RefCav lock attempts (attachments 2 and 3), the fastmon channel will still drift away, but then lose resonance once the signal hits +/-13V (the limit of the PZT as set by the electronics within the TTFSS box) before the temploop is able to turn on. I also looked back to a successful FSS lock with the NPRO installed before this one (before the problems with the autolocker started, attachment 4), and the behavior looks much the same as with successful locks with the current NPRO.

It seems that with this NPRO, for some reason, the PZT is frequently running out of range when trying to center on the RefCav resonance before the temploop can turn on to help, but it sometimes gets lucky. Jason and I took some time familiarizing ourselves with the autolocker code (written in C and unchanged in over a decade) to give us a better idea of what it's doing. At this point, we're still not entirely sure what about this NPRO is causing the PZT to run out of range, but we do have some ideas of things to try during a maintenance window to make the FSS lock faster:

• Reduce delay turning on temploop (state 2; currently 1.0 sec)
• Increase wait time for mode to be centered (state 1; currently 0.1 sec; LLO uses 5.0 sec)
• Allow more voltage to PZT (requires using newer version of TTFSS box)
  • Current TTFSS limits PZT voltage to 13V-15V max
  • HV capable of supplying up to 65V
  • PZT capable of running up to 65V-100V

Starting from the somewhat strange RM1 spectra I saw earlier today (alog ), I have been looking at HAM1-related things.  I don't think this is a strong correlation, and maybe this just means that the HAM1 FF is doing what it should be, but it seems that the TTL4Cs on HAM1 are qualitatively different between times of good range and poor range.  However, the confusing thing is that the L4Cs seem bad at times when the range is good, which means that I don't really understand how they could be causing our troubles.  Also, other times seem to not have this inverse pseudo-correlation.  So, I'm not so sure that this is a sign of our troubles, or just something totally unrelated.

Sheila found that H1:TCS-ITMX_CO2_ISS_CTRL2_OUT_DQ stepped up from -0.2 to 0 on 2024/12/05 21:40:07 UTC (13:40 PST) Plot attached. This is of interest as this channel has ben a witness to our noisy/low range periods in DARM but is not connected to anything. I see no reason for this step up, the only person in the LVEA at the time was Robert setting up VP measurements (near HAM3 not CO2X) 81628, the CO2 laser remained l locked and we were not touching the CO2 chiller around the time 81634.

Since this step up, this channel has not been a good witness of our DARM noise, maybe the cable wasn't grounded and something changed to ground it on 12/05. Plot of it being a witness to the noise on 12/02 and not on 12/11.

Before and after the step up, this CO2 channel is still a witness to the CO2 rotation stage moving, attached. Both CO2X and CO2Y ISS CRTL2 channels see the rotation stages move, some crosstalk in chassis? CO2Y signal is orders of magnitude larger.  Jason was looking into these channels and the chassis

TITLE: 12/12 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: We stayed locked the entire shift, over 23 hours as of 00:30 UTC. Lost of investigations into the range drops today, range has been steady since the last occurance ~14hours ago. If the drop happens again there's a plan to investigate out of observing alog81774.
LOG:                                                                                                                                                               

• The VAC prep dust monitor is offline, I also reset the DM alarm levels that were reset yesterday following E1600132
• 18:24 UTC Tyler and the crane crew started down the gravel road to the FCES, into the enclosure at 18:26 and they left at 18:35 UTC (tagging DetChar)
• ~22:00 UTC I noticed the SQZ_PMC GRD notifying about the PMC VOLTs being low
  • 22:05 UTC the SQZer lost lock and dropped us out of Observing
  • The FC struggled to relock, FC2 P was very misaligned. I brought it closer to where it was when we last locked until it was close enough for FC-ASC to take over
  • 22:22 UTC Observing
• After lots of trending I think I found another ALS channel thats witnessing something related to the range drops, H1:ALS-Y_WFS_B_DC_YAW_OUT16 but I imagine its related to the 2 ALS_Y_WFS_B_DC channels IDed in alog81764
• 23:40 UTC BSC3 vac pressure glitch at PT132

I can't find the right alog for directions, so here is an easily searchable set of directions for HAM1 FF.

There is a master switch for the HAM1 feedforward, but it can sometimes cause problems to slam it on and off that way. Instead, you can ramp the input to the feedforward down to zero. From sitemap:

SEI > ISI Sensor Config > [middle of the screen, see attachment] HAM1 ASC FF > L4CINF

This opens a filter bank page with four filter banks. They each have a gain of 1 and a ramp time of 20 seconds. Set all of these gains to zero to turn off the input to the feedforward. Ramp them back to 1 to engage.

As several of us just talked about in the control room, if The Noise is happening when folks are on site and there's a plan of a thing to try, please feel free to drop Observing to check.  I think we'll keep the list of things to try elsewhere more dynamic than the alog (probably the LHO commissioning google doc).  Some examples of things that we're thinking of right now are (a) turning off the HAM1 FF, or (b) walking (gently) around electronics racks (CER, EX ESD driver area) to see if we can hear any electronics 'whining' or otherwise going bad.

Please do send me a mattermost message, which should audibly ping my phone, but this is causing such problems for our data quality that there is no need to wait for a response from me before trying something.