TITLE: 09/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

- H1 had issues today aligning our Y arm, more info from Ryan's alog here

- TJ and Tony did some troubleshooting by walking the Y arm suspensions in combination with turning off any egregious output signals coming from the ALS Y WFS once they turned on, this seemed to do the trick

• Alignment was still poor on relocking, but at least I now have good flashes to work with so I will be running an initial alignment and trying to relock again afterwards
  • Had to touch SRM a bit to align SRC during the IA, but otherwise it went completely automated

- Upon relocking - the good news is that ALS Y caught fairly quick but I am noticing that ALS Y WFS DOF 1 Y is still swinging a lot more than the rest of the WFS

• Other than that relocking went very smoothly!

- Back to NLN @ 1:05 UTC/OBSERVE @ 1:23

• PI 31 ringup @ 1:12, damped by guardian
• EX saturations @ 1:21/3:25
• 2:04 - Superevent S230922g
• 4:07 - Superevent S230922q

LOG:
No log for this shift.

Following issues during the day shift trying to get a stable Y arm on ALS, H1 looks to have fully recovered and is back in observing. Systems appear stable and ground motion is low.

I noticed in the most recent cps famis task that there is a very sharp peak in a number of ITMY ISI sensors. It's visible in a number of dofs, but trolling through the spectrograms on the summary pages, it seems like it's not always visible, so I never noticed it on the wall foms. This peak seems to be very coherent with at least DARM and CHARD when it is visible, so it's I will try to figure out a configuration that will let us avoid it. It hasn't been visible today,so I haven't been able to try anything, but suspect that using a blend that doesn't use the sensor that I suspect is failing. This makes 3 "bad" l4cs in the IFO all ST1 H1 L4Cs, I don't have an explanation, but that is very suspicious.

As far as which sensor, I suspect it is the St1 H1 L4C. Looking at tranfer functions among the various mostly co-aligned sensors, the H1 L4C consistently has diferent behaviour from the other sensors, particularly around and below the L4C pendulum frequency. 

First image, shows the corner ITMY ground sensor and the ITMY St1 Y T240. Red is the ground sts, blue is the ITMY St1Y, green is the ITMX St1Y. The peak at 1.32hz on ITMY is more than a factor of 10 above the ground motion, but this was at 9 utc on the 20th, asds don't show the peak right now.

Looking at the some of the ground and ISI motion blrms channels this peak has been coming and going over the last couple of weeks. Second image compares the ground, itmy st1, bs st1 and itmx st1 y 1-3hz blrms over the last couple weeks. It certainly seems like about the time that the ITMY st2 h2 cps started glitching, the ITMY ST1 H1 L4C started to intermittently get noisy. Seems like a reasonable explanation is the L4C is failing some way internally and the multiple trips from the CPS glitching made that get worse.

Third tf plot shows the co-located passive sensor tfs for the L4Cs, the dashed lines are the H1 l4c to H1 cps and X1 T240 tfs, the other traces are the H2 and H3 l4cs. Red, blue and green should all be pretty similar and look something like the typical 1hz seismometer response, but the dashed red trace looks like something has changed the low frequency response of the H1 L4C. There's a notch in the blue and green traces, I think because the peak in the H1 sensor is so loud it's all the other sensor can see. The brown, light blue and pink traces are the local L4C to CPS tfs. Kind of similar, kind of different, the low frequency reponse of the H1 sensor is different from the other 2 sensor pairs, and but the peak doesn't show up as much in the other sensors because of CPS noise or something. Need to think about that.

Fourth asd plot shows the peak shows up the most in the H1 L4C, less visible in the other L4Cs.

Last tf plot shows the local CPS to T240 tfs. Peak is equally visible in all these sensor pairs, transfer functions all look generally the same.

Probably the best shot for fixing this at the moment is switching to blends that don't used the H1 L4C, but the peak needs to be visible to test that and it hasn't been cooperating so far today. The channel H1:ISI-ITMY_ST1_FFB_LOG_Y_1_3 has been the easiest to trend witness so far, peak is active when that channel hits ~1.

TITLE: 09/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Austin
SHIFT SUMMARY: H1 recovered after this morning's h1sush7 crash and was observing for most of the morning. Since then, we've struggled to align the Y arm well enough to lock. Recovery is ongoing.

• 15:47 Resumed observing following recovery of h1sush7, see alog 73026 for details
• 17:26 Lockloss - alog
• Started relocking, several locklosses before we could move off of ALS from Y arm losing lock
• 20:13 Have been working to align Y arm by hand
• 20:36 Trying initial alignment since I've moved ITMY by this point
• 21:50 EQ mode activated from 6.1 Vanuatu quake
  • 21:55 Back to calm
• Eventually I was able to get Y arm flashes good enough to lock the cavity and finish initial alignment
• 21:53 Alignment finished, Y arm transmission is still as bad as before
• 22:01 EQ mode activated from 5.7 Japan quake
  • 22:21 Back to calm

LOG:

TITLE: 09/21 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 7mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.22 μm/s 
QUICK SUMMARY:

- IFO is currently DOWN troubleshooting some alignment issues coming this morning's failed IO chassis

To continue trying to understand how OM2 is improving range, I compared some recent sqz / no-sqz times with hot/cold OM2. So far, I still see the optical gain reduction with hot OM2 (i.e. cold OM2 is better >kHz), and the improvement in low-frequency noise with hot OM2 (~30-300 Hz?).

• Sept 2023, different OM2 temps for no-sqz / sqz darm (maybe most interesting plot).
  • The bottom plot shows the ASD ratio of DARM with cold/hot OM2, where cold/hot ratio > 1 implies that DARM noise with OM2 hot is better than with it cold (i.e. measured darm with hot om2 is less noisy there than with cold om2).
• Same plot but including the same traces from July (adding to LHO:71533) ****
• And sqz/no-sqz at the various OM2 temps.

****Caveat regarding the calibration: I am simply plotting GDS-CALIB_STRAIN_NOLINES for these various times, but this is not the most accurate for the traces before September, as I did not use any corrections to GDS for those times.

I'm interested in this because OM2 was re-heated this week LHO:72967 after having been off for some time, possibly off since ~8/29. If we assume OM2 was cold on bad sqz night following 9/10, LHO:72796, and OM2 was hot & thermalized for today's 9/21 no-sqz time due to IO chassis power issues impacting HAM7 ISI LHO:73026 (reasonable to assume OM2 is hot; the remaining thermister 1 shows a plateau in temperature today) -- then we have some long quiet stretches to compare. For the sqz subtraction (ie, sqz dB as a function of frequency), last done with hot om2 on 8/2 for many sqz angles 72565, these long quiet sqz/no-sqz stretches are super useful. I'm still working on more IFO modelling to be more confident in quantum noise parameters as a function of OM2.

For reference, Camilla helped find the following sqz quiet times, building on the OM2 times used before in LHO:71533. A full dictionary of times used for plotting is attached.
'cold OM2 4':{  # bad sqz night @ https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72796
                     'no sqz': {
                         'gps start': 1378445718, 
                         'gps stop' : 1378446918
                     },
                     'FDS': {
                         'gps start': 1378440000,
                         'gps stop' : 1378441800
                     }},
'hot OM2 4':{   # OM2 got cold sometime https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72967
             # these hot sqz/no-sqz times are from 9/21 when ham7 isi tripped for several hours, https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=73020
                     'no sqz': {
                         'gps start': 1379340154,
                         'gps stop' : 1379345854
                     },
                     'FDS': {
                         'gps start': 1379324600,
                         'gps stop' : 1379328200
                     }},

Lockloss @ 17:26 UTC - ISS secondloop looks to have saturated.

IMC_LOCK guardian noted at 17:26:05 "1st loop is saturated, and open" and moved to OPEN_ISS to open the secondloop. Upon entering the OPEN_ISS state, IMC_LOCK jumped to DOWN (I'm unsure exactly why) and after turning off the secondloop boosts, the lockloss occurred. I'll continue looking into this.

Thu Sep 21 10:23:37 2023 INFO: Fill completed in 8min 33secs

Travis confirmed a good fill curbside.

Code did not run the fill at the scheduled time of 10:00:00 due to time bug. I rescheduled for 10:15:00, at which time it ran correctly.

Ryan, Rahul, Fil, Dave:

All h1sush7 models stopped running at:

PDT: 2023-09-21 03:48:52.000000 PDT
UTC: 2023-09-21 10:48:52.000000 UTC
GPS: 1379328550.000000
 

From this time onwards H1 was still in lock with a depressed range of ~130MPc and was out of OBSERVE. h1seih7 SWWDs were tripped.

Recovery process was:

Stop models, fence h1sush7 from the Dophin fabric, reboot h1sush7.

When h1sush7 came back, I verified that the IO Chassis could not be seen. I then fenced and powered the front end down.

Fil went onto the mech room mezzanine and verified that the SUS side of the Kepco dual-power-supply had tripped (SEI side was OK). He powered the IO Chassis back on, I powered h1sush7 computer and all came back correctly.

I untripped the SWWDs to get h1seih7 driving again, Ryan and Rahul recovered the SUS models.

Famis 26458
HEPI Pump Trends for the last 45 days are attached.
H1:HPI-PUMP_L0_CONTROL_VOUT has seen a change of 8[units], in the last 3 days. Im not sure if movement of 8 is unreasonable or not. Also what are the units of these channels?
 

H1EPICS_HWS.ini has 88 ITMY channels, all are connecting except the following 6:

H1:TCS-ITMY_HWS_BEAM_POS_X
H1:TCS-ITMY_HWS_BEAM_POS_Y
H1:TCS-ITMY_HWS_CO2_POS_X
H1:TCS-ITMY_HWS_CO2_POS_Y
H1:TCS-ITMY_HWS_RH_POS_X
H1:TCS-ITMY_HWS_RH_POS_Y
 

Detchar, please tell us if the 1.66Hz comb is back.

We changed the OM2 heater driver configuration from what was described in alog 72061.

We used a breakout board with jumpers to connect all OM2 thermistor readback pins (pin 9, 10, 11, 12, 22, 23, 24, 25) to Beckhoff at the back of the driver chassis. Nothing else (even the DB25 shell on the chassis) is connected to Beckhoff.

Heater voltage inputs (pin 6 for positive and 19 for negative) are connected to the portable voltage reference powered by a DC power supply to provide 7.15V.

BTW, somebody powered the OM2 heater off at some point in time, i.e. OM2 has been cold for some time but we don't know exactly how long.

When we went to the rack, half of the power supply terminal (which we normally use for 9V batteries) was disconnected (1st picture), and there was no power to the heater. Baffling. FYI, if it's not clear, the power terminal should look like in the second picture.

Somebody should have snagged cables hard enough, and didn't even bother to check.

Next time you do it, since reconnecting is NOT good enough, read alog 72286 to learn how to set the voltage reference to 7.15V and turn off the auto-turn-off function, then do it, and tell me you did it. I promise I will thank you.

J. Oberling, A. Jennings

With a relatively light maintenance window and a chance for some extended Laser Hazard time, we finally completed the installation of an old OpLev ECR, FRS 4544.  Austin and I removed the existing 10m single-mode fiber and installed a new, 3m armored single-mode fiber, and placed the laser in a "thermal isolation enclosure" (i.e. a Coleman cooler).

To start, I confirmed that there was power available for the new setup; the OpLev lasers are powered via a DC power supply in the CER.  I read no voltage at the cable near the ITMx OpLev, so with Fil's help we traced the cable to its other end, found it unplugged, and plugged it in.  I confirmed we had the expected voltage, which we did, so we moved on with the installation.  We had to wait for the ITMx front end computer to come back up (had tripped as part of other work), so while we waited Austin completed the transition to Laser Hazard.  We took a picture (1st attachment) of the ITMx OpLev data (SUM counts, pitch and yaw readings), then powered down the laser.  We placed the laser in the cooler and plugged in the new power supply; laser turned on as expected.  We then installed a Lexan viewport cover and removed the cover from the ITMx OpLev transmitter pylon.  The old 10m fiber was removed, and we found 2 areas where the fiber had been crushed due to over-zealous strain relief with cable ties (attachments 2-4; this is why we originally moved to armored fibers); I'm honestly somewhat surprised any light was being passed through this fiber.  We installed the armored fiber, being careful not to touch the green camera housing and to not overly bend the fiber or jostle the transmitter assembly, and turned on the laser.  Unfortunately we had very little signal (~1k SUM counts) at the receiver, and the pitch and yaw readings were pretty different.  We very briefly removed the Lexan cover (pulled it out just enough to clear the beam) and the SUM counts jumped up to ~7k; we then put the Lexan back in place; we also tried increasing the laser power, but saw no change in SUM counts (laser already maxed out).  This was an indication that we did manage to change the transmitter alignment during the fiber swap, even though we were careful not to jostle anything (it can happen, and it did), and that the Lexan cover greatly changes the beam alignment.  So we loosened the locking screws for the pitch and yaw adjustments and very carefully adjusted the pitch and yaw of the launcher to increase the SUM counts (which also had the effect of centering the beam on the receiver).  The most we could get was ~8.3k SUM counts with the Lexan briefly removed, which then dropped to ~7k once we re-installed the transmitter cover and completely removed the Lexan (no viewport exposure with the transmitter cover re-installed).  We made sure not to bump anything when re-installing the transmitter cover, yet the SUM counts dropped and the alignment changed (the pitch/yaw readings changed, mostly yaw by ~10 µrad).  Maybe this pylon is little more loose than the others?  That's a WAG, as the pylon seems pretty secure.

I can't explain why the SUM counts are so much lower; could be the difference between the new and old fibers, we could have really changed the alignment so we're now catching a ghost beam (but I doubt this, we barely moved anything).  Honestly I'm a little stumped.  Given more time on a future maintenance day we could remove the receiver cover and check the beam at the QPD, but as of now we have what appears to be a good signal that responds to pitch and yaw alignment changes, so we moved on.  We re-centered the receiver QPD, and now have the readings shown in the 5th attachment; ITMx did not move, it stayed in its Aligned state the entire time.  This is all the result of our work on the OpLev.  We'll keep an eye on this OpLev over the coming days, especially watching the SUM counts and pitch/yaw readings (looking for drift and making sure the laser is happy in its new home; it is the oldest installed OpLev laser at the moment).  The last few attachments are pictures of the new cooler and the fiber run into the transmitter assembly.  This completes LHO WP 11422 and FRS 4544.