https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1385827768

Lockloss from commissioning activities alog74604

The chillers were resetting their set points based on outdoor air temperature, so the last week of cold weather followed by several days of warmer weather was causing issues for the PID loops in the programming. I disabled the outdoor air temperature reset feature on the chillers to keep them from deviating from their initial set points. 

TITLE: 12/05 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 1mph Gusts, 0mph 5min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.83 μm/s
QUICK SUMMARY: Coming back up from a lock loss, almost there but maintenance is starting.

Workstations were updated and rebooted.  This was an OS package update.  Conda packages were not updated except on the new workstations, cdsws22, cdsws23, and cdsws24, emacs was removed from the conda environment.  emacs is still available as an OS package.

TITLE: 12/05 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

IFO is in NLN and OBSERVING (~11 hr lock).

1:23 UTC - Alarms going off on the EX and EY Chillers despite systems looking normal. This is potentially as a cause of a recent chiller temp issue, though the alarms persist since the temperature recovery. Keeping an eye on it (no alarms from my workstation). Due to the temperature fluctuating at the alarm threshold degree for the EX chiller supply, Dave changed the threshold from 50F to 55F (alog 74587). The same problem happened at EY and called Richard multiple times. The temperature is sitting between 49F and 50F (and triggering alarms it seems as it crosses). Dave later changed this to 55 too (alog 74589)

1:45 UTC - Jenne finished touching PR3 to address the clipping issues (alog 74581)

IY CHILLER 36 Hr Trend (screenshots).

Per alog 74563 (and other associated alogs), the screenshot shows a 36 hr trend of IY Chiller flow rates and process pressures. There don’t seem to be significant changes to the trends shown with the exception of IY CO2 Flowrate (green). There is a sharp peak that occured at around 6:45 UTC for only 2 seconds. The counts during this peak iincreased from 2.9 to 4.7 (~160%). The second and third screenshots illustrate how this peak looks compared to surrounding noise. There were no other significant deviations (from what I could tell). Tagging TCS.

LOG:

IFO is in NLN and OBSERVING (about 7 hours).

Nothing else of note.

The EX chiller is running at a slightly higher temperatures since 15:00 PST this afternoon. I has been occassionally exceeding its alarm level of 51F causing cell phone alarms. Richard has asked me to increase the alarm level.

The alarm level is now 55F as of 18:35, 24 hour trend is attached.

old Channel name="H0:FMC-EX_CY_H2O_SUP_DEGF" low="32" high="51" description="EX CHILL-SUP-H2O"
new Channel name="H0:FMC-EX_CY_H2O_SUP_DEGF" low="32" high="55" description="EX CHILL-SUP-H2O"

At today's commissioning meeting, we decided that we would very, very slowly move PR3 while in Observe, so see if we could further reduce clipping in the PRC / in the POP pickoff path. 

To check that I could move the PR3 sliders while in Observe, I clicked the 'MASK' button in the SDF to ALL (so that I could see which two channels are not monitored in the PR3 SDF).  Indeed the PR3 sliders are not monitored, but just checking that brought us out of Observe briefly.  The Operator brought us back to Observe a few seconds later.  Naoki and I also unmonitored the _TRAMP channels for PR3's sliders, but that did not take us out of Observe. 

Set the TRAMPS to 10 sec from nominal 2 sec.

Yaw steps of -0.0001 are okay. -0.0003 start to see things in the camera signals, so seem too big for Observing.  Kept taking steps of -0.0001 every ~30 sec.  Moved a total of -0.002 PR3 yaw. Now sitting for a few minutes of quiet, starting at 5 Dec 2023 00:00:00 UTC for 5 mins.

Naoki looked, and we can't see any yaw motion with my moves, although there's a teeny bit of pit motion.  So we're going to try again the steps of -0.0003.  Now, they seem to not cause a glitch, so the one earlier was probably a coincidence.

Increased step size to -0.001, seems okay. 

Increased step size to -0.003, seems okay.  Got to PR3 yaw offset slider of 152.000, so going to start some mins of quiet at 00:37:00 UTC. Quiet ended at about 01:10:00 UTC, although in between Robert went to the CER twice to turn off, then back on an AC unit.

Seems like the things I was worried about being glitches are just peaks of the CER ACs (but not the one that has been a problem in the past, since Robert turned that off for a few mins and we didn't see a change). So, bigger step size of -0.01 is going okay.

I'm stopping for now.  We've moved about 0.2 urad in yaw, continuing in the direction that Sheila did earlier today (Sheila did a little more than 1urad when in PRMI only).  Naoki is making a plot showing that the coherence between the LSC and ASC POP sensors is basically the same now as after Sheila's move earlier. So, I think we're at least not on the edge of clipping, and that it's a better use of time to do more moves during a PRMI or DRMI lock, when we can move faster and see about finding the center of our clipping aperture.

Recent monthly Fscan spectra show a comb at multiples of 0.996785 Hz, in approximately the region 20-100 Hz. Subsequent investigation shows that it probably appeared between Sept 21 and Sept 24, although the exact date is difficult to tell.

Further details:

• The comb is clearly visible in monthly spectra, which show its appearance in September [figure 1-2].
• It's somewhat visible in weekly spectra (although generally too weak for the combfinder to detect), which shows its appearance during the week of Sept 20 [figures 3-5]
• It's essentially invisible in daily spectra unless you know exactly what to look for. Fortunately, once the exact frequencies are known, the median height of the comb teeth can be tracked. It's not possible to confidently determine a start date, but it's likely after Sept 20 and on or before Sept 24. [figures 6-9]

Note that there was a comb in O3 with spacing 0.996806 Hz (which, on inspection of the O3 spectrum, seems to have a double-peak structure). Although they are very close, the new comb comb does not precisely align with the O3 comb, nor with its second peak.

TITLE: 12/05 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:

Lockloss @ 18:15
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1385748973

Relocking
intervined in ALSY green arm because increase flashes

Sheila asked to hold in PRMI Locked to investigate clipping of PR3/PR2.
Sheila touched up PR3& PR2, We then tried to lock and got a lockloss @ DRMI_1F 19:10UTC

Relocking got all the way up to NLN @ 20:00 UTC

Lockloss 20:04 UTC
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1385755476

Back to Observing!!! @ 20:55

Jennie Is still moving PR3 very slowly and delicately.

LOG:

TITLE: 12/05 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 4mph 5min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.61 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING

Attached shows the comparison of ASC-POP_A_NSUM (one of the QPD on the sled in HAM3) and LSC-POP_A (LSC sensor in HAM1) before (references in DTT) and after (current) the PR3 realignment, both in full lock. To the left is the trend showing the PR_GAIN (which comes from LSC POP_A in HAM1) at the top. I chose 2023/11/30 070000 UTC as pre-adjustment and 2023/12/02 090000 UTC as post-. Note that Sheila has made yet another adjustment after this "post-adjustment" time, but that's not included in this alog.

If you compare ASC-POP before and after (green and red on the right bottom) the noise level was about the same. If you compare LSC-POP before and after (brown and red on the right bottom), clearly there was a lot of extra noise before the realignment. This suggests that there was extra clipping in the LSC POP path. After the adjustment, ASC and LSC POP agreed with each other as you can see from the coherence (red trace on the right top), so the extra clipping in the LSC POP path was gone. Note, that does not necessarily mean that there is no clipping inside PRC which will appear as common clipping on both ASC and LSC POP sensors.

For the difference of these paths in HAM3, see https://dcc.ligo.org/LIGO-D1000339. LSC beam is the PR2 transmission of the back-propagating beam (coming from PR3), and ASC beam is the transmission of the forward-propagating beam (coming from PRM). The only common component in these paths is the fixed 3" optics right behind the PR2.

FAMIS 26484

T240 centering:

Averaging Mass Centering channels for 10 [sec] ...
2023-12-04 14:07:32.265709

There are 18 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 1 DOF X/U = 0.382 [V]
ETMX T240 1 DOF Z/W = 0.35 [V]
ETMX T240 2 DOF X/U = -0.861 [V]
ETMX T240 2 DOF Y/V = -1.262 [V]
ETMX T240 3 DOF Y/V = 0.311 [V]
ITMX T240 1 DOF X/U = -0.891 [V]
ITMX T240 1 DOF Y/V = 0.385 [V]
ITMX T240 1 DOF Z/W = 0.5 [V]
ITMX T240 2 DOF Z/W = 0.302 [V]
ITMX T240 3 DOF X/U = -0.885 [V]
ITMY T240 3 DOF X/U = -0.482 [V]
ITMY T240 3 DOF Z/W = -1.246 [V]
BS T240 1 DOF Y/V = -0.362 [V]
BS T240 3 DOF Y/V = -0.304 [V]
BS T240 3 DOF Z/W = -0.455 [V]
HAM8 1 DOF X/U = -0.345 [V]
HAM8 1 DOF Y/V = -0.307 [V]
HAM8 1 DOF Z/W = -0.541 [V]

All other proof masses are within range ( < 0.3 [V] ):
ETMX T240 1 DOF Y/V = 0.259 [V]
ETMX T240 2 DOF Z/W = -0.063 [V]
ETMX T240 3 DOF X/U = 0.264 [V]
ETMX T240 3 DOF Z/W = 0.237 [V]
ETMY T240 1 DOF X/U = 0.146 [V]
ETMY T240 1 DOF Y/V = 0.207 [V]
ETMY T240 1 DOF Z/W = 0.273 [V]
ETMY T240 2 DOF X/U = 0.011 [V]
ETMY T240 2 DOF Y/V = 0.243 [V]
ETMY T240 2 DOF Z/W = 0.11 [V]
ETMY T240 3 DOF X/U = 0.297 [V]
ETMY T240 3 DOF Y/V = 0.183 [V]
ETMY T240 3 DOF Z/W = 0.223 [V]
ITMX T240 2 DOF X/U = 0.197 [V]
ITMX T240 2 DOF Y/V = 0.298 [V]
ITMX T240 3 DOF Y/V = 0.199 [V]
ITMX T240 3 DOF Z/W = 0.178 [V]
ITMY T240 1 DOF X/U = 0.127 [V]
ITMY T240 1 DOF Y/V = 0.083 [V]
ITMY T240 1 DOF Z/W = -0.038 [V]
ITMY T240 2 DOF X/U = 0.087 [V]
ITMY T240 2 DOF Y/V = 0.194 [V]
ITMY T240 2 DOF Z/W = 0.107 [V]
ITMY T240 3 DOF Y/V = 0.095 [V]
BS T240 1 DOF X/U = -0.154 [V]
BS T240 1 DOF Z/W = 0.112 [V]
BS T240 2 DOF X/U = -0.075 [V]
BS T240 2 DOF Y/V = 0.03 [V]
BS T240 2 DOF Z/W = -0.121 [V]
BS T240 3 DOF X/U = -0.171 [V]

Assessment complete.

STS Centering:

Averaging Mass Centering channels for 10 [sec] ...

2023-12-04 14:11:47.261653
There are 2 STS proof masses out of range ( > 2.0 [V] )!
STS EY DOF X/U = -4.112 [V]
STS EY DOF Z/W = 2.787 [V]

All other proof masses are within range ( < 2.0 [V] ):
STS A DOF X/U = -0.545 [V]
STS A DOF Y/V = -1.003 [V]
STS A DOF Z/W = -0.369 [V]
STS B DOF X/U = 0.525 [V]
STS B DOF Y/V = 0.91 [V]
STS B DOF Z/W = -0.547 [V]
STS C DOF X/U = -0.509 [V]
STS C DOF Y/V = 0.793 [V]
STS C DOF Z/W = 0.37 [V]
STS EX DOF X/U = -0.208 [V]
STS EX DOF Y/V = 0.015 [V]
STS EX DOF Z/W = 0.104 [V]
STS EY DOF Y/V = 0.201 [V]
STS FC DOF X/U = 0.278 [V]
STS FC DOF Y/V = -0.912 [V]
STS FC DOF Z/W = 0.72 [V]

Assessment complete.
 

Because Tony was relocking, I asked him to stop at PRMI while I walked PR3 further in the direction that Corey and I walked it Friday (74535).  This increased POP18, but it made ALS COMM beatnote worse. 

The second screenshot shows these trends going further back, the move of PR3 on Nov 7th is shown by the cursors. It looks like we lost some PRG on that day, and readjusted the COMM beatnote for a PR3 alignment that was giving us worse PRG. This was stable for a few weeks but some drift in the last 2 weeks has made the PRG even worse.

These are some trends that Tony and I made over the weekend that would be useful to have in the log.

The first trend shows a couple of months, our PRG has been degrading for about 30 days.  This looks to be correlated with the PR3 optical lever, but that optical lever (along with many other alignment and vertical sensors) correlates with outdoor temperature.  The COMM beatnote depends on the PR3 angle, this was re-adjusted on the table 74065 a month ago, but also looks to have been dropping over the last few weeks.  The second screenshot shows that move of PR3 (and PR2) that Corey and I did on Friday night improved the PRG back from 40 to 48 (at the max of the thermal transient), improved the COMM beatnote slightly, and brought the PR3 optical lever partially back to where it was a few months ago. 

The last attachment shows some sensors related to PR3.  The IPS and CPS say that HAM2 hasn't moved in yaw.  The osems and optical levers do not agree about what has hapened with PR3, the optical lever says that we have had a drift in the positive yaw direction, and that we partially reversed that on Friday night.  The top mass damping says that we had a negative yaw drift, and that our move Friday night went further in the direction of the drift.

The last attachment shows some similar sensors for SR3, which doesn't have as large of a drift.

Mon Dec 04 10:03:51 2023 INFO: Fill completed in 3min 49secs

Gerardo confirmed a good fill curbside.

Now that it has warmed up outside I have lowered the trip temps to -80C.

This evening as soon as Corey started his shift he found several of the violin modes were rung up (see screenshot of the darm), which is when he texted me to look into it. Most of the modes were coming down slowly with nominal settings, however I had to tweak the settings of the following two modes, the details are given below,

ITMY05/06: FM1+FM8+FM10 Gain -0.01 (will ramp it up later)

ITMY08: FM1+FM5+FM10 Gain= +0.1 (earlier I tried a phase of +60degree and that seems to be making this mode worse).

The above settings are not in the lscparams yet, will test them out for few locks.