Camilla, Jonathan, Erik, Dave:

Erik went to EY and took a look at the h1hwsey computer. It had a stack trace display on its console which does not appear to give any clear indications of the crash.

Subsequent computer reboots and power cycling (including power cord pull out) would not boot the computer. The FAIL red LED is on, the power green led is off, nothing appears on the local console.

Camilla says we can run for the rest of the week without ETMY HWS.

I have green'ed up the EDC on the CDS Overview, where GREEN=86 disconnected channels.

Since last weekend ITMY mode05/06 were rung up and the nominal settings were not able to damp them. During that time the microseism and ground motion was higher than nominal, also LVEA temperature was slightly elevated. On Monday I found a new setting which worked all day and during evening shift (as confirmed by Ryan C). However it stopped working this morning. Hence, I again tried the nominal settings and it seems to be working for now. Given below are the details,

Settings working today (Nov 01 2023): FM6 + FM8 + FM10 Gain -0.01 (will slightly ramp it up later, although we lost lock while I was writing this alog, nevertheless I will try again) - see plot attached.

Settings which seems to be working on Monday (Oct 30 2023): FM6 + FM10 Gain 0.01

I will keep a watch on them before committing it in the lscparams.

Wed Nov 01 10:13:20 2023 INFO: Fill completed in 13min 16secs

Jordan confirmed a good fill curbside.

We were only locked for 1:08, lockloss at 19:29UTC. ITMX was the first suspension to saturate right before the LL and there was a 500Hz oscillation in LSC-DARM which was probably the violin modes.

We recovered NLN at 16:10UTC after damping violins for about a half hour and Observing at 16:46UTC, VIcky touched up the SQZer after it was having issues and Erik went out to investigate the EY HWS computer issue but was not able to resolve it as it seems dead. We have comissioning planned for 12pm - 4pm PST, 19:00 - 23:00 UTC. GDS was also down but has been brought back.

Jordan, Gerardo

Yesterday during Tuesday maintenance (10/31/23), we were able to install the final two of the new 150 l/s ion pumps on FC-B section, B4 and B5 crosses. The Ion Pump/Tee/Angle valve assembly was pre-built in the staging building and then pumped down and leak checked. This assembly was stored under vacuum and then brought to the LVEA.

We closed FCV-3 (BSC3), FCV-4, FCV-5, and FCV-6 (HAM8) to isolate the B4 and B5 crosses. Then closed the -Z axis GV on the two crosses to isolate the ion pump port. We then vented the ion pump assembly with N2 and removed the angle valve/6" zero-length reducer on the cross, and installed the ion pump on the 6" CF port. A genie lift was used to lift/hold the ion pump while the connections were made.

Once installed, we used the leak detector/small turbo to pump down the assembly, and then helium leak tested the one CF connection that was made. There was no detectable signal above the helium background of 9.5E-11 Torr-l/s.

The ion pump was powered on locally, and quickly dropped to ~2mA. The -Z gate valve remains closed, and the rest of the FCT gate valves were reopened once the ion pump was leak checked and powered on. This concludes the installation of the ion pumps, high voltage conduit and communication cables to follow in the following weeks.

TITLE: 11/01 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 0mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:

• The IFO has not relocked since maintenaince yesterday
• Looks like an initial alignment was finished about an hour ago by Jenne alog73892, just locked PRMI
• We just had an earthquake roll through an hour ago
• 86 channels showing no connection, alog73885

TITLE: 11/01 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
Inherited H1 during Initial Alignment, stuck on ACQUIRE_XARM_IR.

Tried locking the IFO just to see where H1 will stumble.
Tried Initial Alignment and got stuck  at Acquire_XARM_IR.
Tried Man Alignment and got stuck in the same spot.

Found my own alog about a very similar situation:
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=69497

Started reading Troubleshooting Docs-
Checked TRX OPD , Trending PR2.
reverting PR2 to GPS time : 1382733517
reverted xarm to  gps time 1382733517
Retouched up ALS x arm in green arms manual tried to do an initial alignment again and got stuck at
ACUIRE_XARM_IR again

~ 9:47 UTC Reverted the PRC PRM, PR1,PR2,PR3 to 1382733517
Success!! INITIAL ALIGNMENT moved forward!!!
INITIAL ALIGNMENT completed!
Locking Got Stuck at ALS and Find IR. So running Another initial Alignment. Perhaps moving PR Cavity sliders half way though Initial alignment leads to a bad Alignment, with ALS? Perhaps my movement of PR3 was too much?
 
Still Cant get past Find IR After another IA.

Jenne had popped in to Matter most and Given me a clue to take notes on where PR3 was.
and PR3 and what I was setting it to.
Notes:
PR3P After: -117.4
PR3Y After: 155.3
PR3P before: -116.2
PR3Y Before : 152.1

I'm exausted and spinning my wheels. I think this might still be a PR Cavity issue. But I can no longer think clearly and feel like I'd make irratic changes at this time in the morning. I've gotta step away and get some sleep.

LOG:
No log

TITLE: 11/01 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

I eventually was able to get the IMC locked (73887) and adjust PR3, but for the past 2.5 hours the detector has been trying to complete an initial alignment.

Trying to lock detector:

- Started initial alignment

    - Got stuck at ACQUIRE_XARM_IR, which I assumed was because it was "Waiting for sus to settle", and it stayed there for almost 30 minutes.

- Took the detector to DOWN and restarted the initial alignment. This time the DCPD values were going down, but very slowly.

- Tony and I left initial alignment and tried to move through normal locking but lost lock at FIND_IR

- Tried going back into initial alignment again, but got stuck at ACQUIRE_XARM_IR again

- Tony and I still troubleshooting - I think it's maybe possible that although I got the IMC locked and aligned within itself, it might not be aligned with respect to the rest of the detector.

LOG:

23:00UTC Maintenance finishing up

23:15 Starting to try to fix the IMC Locking issue

02:11 After messing with pitch and yaw on MC1 and the IMC PZT for a while, I wasn't able to get them back to the values that they were at at 15:11UTC this morning, which is when the IMC was last in the unlocked state. I got them all as close as they could and then I am now trying to get flashes on MC trans by going to the acquire state on IMC_LOCK

03:56 IMC Locked

04:53 Adjusting PR3 pitch and yaw based on 73854
    - Got pitch within range from ~1 week ago and yaw is much closer but not within range. COMM beatnote is ~-17 before initial alignment

05:21 Started Initial Alignment

05:55 After about 10 minutes in initial alignment, ALIGN_IFO had gotten stuck at ACQUIRE_XARM_IR, so I took the detector to DOWN and restarted the initial alignment

06:11 Got stuck again at the same spot

07:13 Took detector out of initial alignment and attempted to lock normally - failed at FIND_IR

I am still working on getting the IMC to lock following today's power supply replacement (73871). Based on Jenne's instructions, I tried to get the IMC-WFS_{A,B}_{PIT,YAW}_OUT16 values back to their values from around 10/31 15:11UTC, which was before the restart and when the IMC was offline, by adjusting the offset on the IMC PZT and MC1, and was able to get much closer, but not great.

I then took IMC_LOCK to ACQUIRE and tried adjusting MC2 and MC3 for the best output, but wasn't able to get the beam to look better (or worse), so with Keita's help I lowered the IMC WFS trigger threshold in an attempt to use the WFS to better the alignment. It has worked a bit (we went from locking on the 00 mode at 30counts on IMC-IMC_TRIGGER_INMON to locking at 62 counts).

Chiller 1 had tripped its main breaker due to a bad condenser motor, which also took out a 30 amp fuse. Bad motor will be replaced when a replacement is available. Chiller 2 is currently in lead position. 

References: alog 73706, 73367

Attached is the comparison of Oct/18/2023 22:30 UTC (current traces, no comb, thermistors cut off from Beckhoff but the heater is still driven by Beckhoff) and Oct/17/2023 22:33:40 (references, with 1.66Hz comb plus 1.235Hz SB to the 1.66Hz comb, thermistors and the heater connected to Beckhoff).

Red circles show 1.66Hz harmonics while blue ones show 1.235Hz harmonics or 1.235Hz SB to the 1.66Hz comb.

1.66Hz comb is visible in OMC QPDs (middle row, though only A is shown here, B is similar) at around 270-280 Hz as well as 177-180Hz, but only in PIT and YAW and NOT in NSUM (which I normalized by the DC value). This strongly suggests that this is either alignment or mode shape change.

We don't see anything at the fundamental 1.66Hz though, maybe the SNR is not enough. OTOH 1.235Hz appears in the current traces as well as references in the middle column. I haven't observed anything like 1.235Hz on the floor in Beckhoff, so I assume that this is NOT from the Beckhoff heater driver DAC. It's unclear if 1.235Hz is length-like (i.e. AM) signal coming from the IFO or if it's also alignment/modal shape change as the signal appears in PIT, YAW and NSUM.

In the second attachment, ASC-AS_A and AS_B DC don't see much except AS_A_DC_YAW sees something at 270.76Hz (top blue). The light power on these should be similar to OMC QPDs but WFS DC doesn't have the ISC whitening filter. ASC-AS_C doesn't see anything (2nd attachment bottom). This is what as AS_C is upstream of OM2 and the imprinting of the noise via the DARM loop shouldn't be strong at 270-280Hz, but note that it only receives 10 times smaller power than OMC QPDs, so the SNR is not as good anyway.

To measure the coupling TF from thermistor current to DARM, I'll route OMCD-DCPD_TEST_EXC to one of the thermistors (hot one at first, as it's mounted on the thermal compression ring which is closer to the electrical loop formed by the heater circuit).

(Sheila mentioned that when OM2 heater was off while the full connection to Beckhoff was still maintained, we didn't see the comb even though 1.66Hz thermistor switching should have been going on. It could be a magnetic coupling (e.g. the single loop electric circuit formed around the compression plate by the heater and its wires generates magnetic field that will push floppy thermistor wires that carry 1.66Hz current?) or maybe dependent on the tightness of the compression ring itself. We can easily distinguish the two by injecting into thermistors and turn off the heating . If the coupling change is instantaneous, it's probably the magnetic field.)

Jenne notes that this changing high frequency noise during the first hours of a lock since Wed 25th Oct (purple trace in 73798) may be caused by the new higher CARM gain 73738, changed on that day.

In 73798, I noted The 4.8kHz noise that suddenly changes ~ 1h40 into NLN,  Jenne suggests this could be aliased down CARM gain peaking. Looking at the 64kHz channel, plot attached, noise disappears from 18.4 to 18.7kHz (v. large peak) and appears at 16.6kHz to 16.8kHz and at 21.1kHz.

We have had a weekend of shorter (~5hour) locks and two locklosses form LASER_NOISE_SUPPRESSION state#575 (73787, 73831), the state this CARM gain is changed. Maybe this gain change has made us less stable, we'll discuss today reverting it.

Summary:

It seems that the supply voltage for thermistors is oscillating, the frequency depends on whether or not the supply is loaded with thermistors (830mHz open to 1.66Hz fully connected to the in-chamber thermistors), the amplitude of the oscillation jumps seemingly randomly, and this is also happening for the unused Beckhoff module for the yet-to-be-installed second T-SAMS unit, all measured at the back of the Beckhoff chassis. (Can't we measure this from Beckhoff itself, without me going to the floor?)

Fil and Fernando quickly set up the same Beckhoff module in the lab and didn't observe this. Could we swap or maybe restart the modules in the chassis?

As of now, Beckhoff cable is disconnected from the back of the heater driver chassis. (I'm applying DetChar-Request tag just so people know, but we're just changing from one no-comb configuration to the other.)

Detaisls:

Since the past findings about OM2 and 1.66Hz comb (alogs 73367, 73233, 72967 72241 and 72061) didn't make sense, I went to the floor and remeasured the comb in the Beckhoff heater output (which goes to the heater driver input) as well as thermistor pins in the back of the heater driver chassis while Beckhoff connection was intact.

Turns out that all of these things share the same frequency but the voltage across thermistor pins was ~3 orders of magnitude larger than Beckhoff heater driver output pins (pin 9 and 19) (the latter were referenced from the driver board ground as it's common mode for both pins). I used a scope on battery and the thermistor voltage was like roughly 1Vpp 1.66Hz rectangular wave (1st pic). Yellow is the voltage across pin10 and 23 (across thermistor 1), blue is pin9 and 22 (thermistor 2) of the DB25 at the back of the driver chassis when the Beckhoff cable was still connected.  Voltage difference seemed to have come from the temperature difference of the thermistors (I disconnected the Beckhoff cable and measured the thermistor 1 and 2 resistance incl. cables to be 7.41k and 4.08k, respectively). When I disconnected the cable from the chassis and just measured the pin10-23 and pin9-22 voltage coming from the cable (picture 2), they were both about 1.2V pp. This is supposed to be the source voltage for thermisters. The frequency slowed down by about a factor of 2 (832mHz) when the thermistors were disconnected.

For your convenience, below is a table of which pins are what (see e.g. E1100530 and D2000212). Note that thermistors themselves only have two pins, therefore supply and readback pins are bundled together in chamber as shown. Supply is presumably a reference voltage supplied through a reference resistor.

Went to the CER, disconnected the cable from the back of the Beckhoff chassis and did the same measurement. Frequency didn't change but the amplitude was much smaller (~280mVpp instead of 1.2Vpp) for a while, but suddenly the amplitude of the thermistor 1 supply changed back to 1.2V (pic 3). Nuts. When the beckhoff cable was reconnected (and the connection to in-chamber thermistor was restored) the frequency went back to 1.66Hz (picture 4).

Picture 5 shows pin 10-23 (thermistor 1 supply) and pin12-25 (thermistor 1 readback, which is not connected to anything). Picture 6 is the same thing but for the unused Beckhoff unit for the second T-SAMS. It's strange that the same thing is happening in two independent units. Picture 7 is the thermistor 1 supply and pin 6-19 (voltage output for the heater driver). It really seems that this is a problem of the supply voltage.

I checked the 24V power strip for the Beckhoff chassis but it was good (pic 8 and 9).

Fil and Fernando set up EL3692, which is the Beckhoff unit used for Thermistors. They didn't observe this oscillation behavior.

I wanted to do some injections into thermistors to see how this couples to DARM but didn't have time.