VACSTAT alarm on BSC3. This is another false alarm due to a sensor glitch.

I have restarted vacstat_ioc.service on cdsioc0 to reset.

I cold booted h1hwsmsr1 after it partially froze so that it could not be logged in to.  The hard drive LED was solid on.

WP 12159

High voltage disabled. Control Chassis powered off. Cable to chamber disconnected at rack/chassis.

Continuity test of pins:

1. Pins A & C tied together
2. Pins B & D tied together
3. Pins E & F tied together

All pins and shell open to chamber ground.

FAMIS26013

No noticeable high frequency noise or other odd peaks. These match up with the plots in last week's report as well.

Wed Oct 23 10:08:19 2024 INFO: Fill completed in 8min 15secs

Gerardo confirmed a good fill curbside.

Here's a few plots looking at the PSL glitches since the start of O4a. 

The first plot is similar to Oli's from 80520, blue shows the monitor at all times and orage shows only for times when the IFO was in a low noise state (600 or more) for the whole minute.  Since the blue trace includes transients from the locklosses there isn't much of an apparent trend looking at this data without masking out locklosses.  From the orange trace you can see that the maximum of the glitches that we have been riding out in low noise has gotten higher recently, after drifting somewhat over the run.

The next plot shows sort of a glitch rate, where I've counted minutes where the FSS fast mon channel goes above 0.6 V as minutes with glitches, and haven't made an attempt to try to count how many glitches are happening each minute. 

Aside: Thanks to Erik and Jonathan for helping me understand why Oli has been having a hard time getting data.  The reason was that gwpy no longer supports gap handling in the environment, as all the scripts in this duty cycle repo were doing.  Now we have to put the gap handling as an argument into gwpy, as shown on line 24  here

Quarterly greasing and rotation of VEA AHU fans was completed yesterday. There is a report that one fan is noisier than normal. This is likely due to a failing bearing. Eric is investigating.

T. Guidry C. Soike E. Otterman

TITLE: 10/23 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 3mph Gusts, 1mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:  NPRO swap continues today. A few dust alarms that quickly dissipated, but no other alarms active.

R. Short, J. Oberling

As written in Jenne's alog here, upon finding the spare PSL NPRO SN1661 does not show the glitching we've been seeing with NPRO SN7974 we begsn swapping the PSL NPRO this afternoon.  We started by removing NPRO SN7974 and our glitch test setup, then installed NPRO SN1661.  We immediately noticed the readbacks in PSL Beckhoff were slightly wrong, meaning we need to adjust the potentiometers on the power supply's internal board that provides these readbacks.  With an injection current of ~0.9 A, which gave an output power of ~32 mW, we roughly aligned the beam to our alignment irises that sit between M02 and M03, using M01 to align to the first iris and M02 to align to the second.  We did remove the PSL EOM and lens L01 from the beam path; the EOM will be reinstalled once we have a mode matching solution, and L01 will have to be changed to a new lens (the current f=167mm lens installed for L01 gives us an uncomfortably small beam in the area of the EOM crystal).  We then set a power meter after M02 and turned up the NPRO injection current until the power meter read ~1.8W; the injection current is 2.144 A.  WP13 (a QWP) and WP14 (a HWP) were adjusted to maximize the power read by the power meter (these 2 waveplates serve as polarization clean up for the naturally slightly elliptical polarization of the NPRO beam).  We started with ~1.806 W and ended with ~1.814 W.  Moving the power meter to the output of the NPRO we measured ~1.86 W, so we're losing roughly 46 mW to wrong polarization.  We then calibrated the PSL Beckhoff reading for the NPRO power monitor PD, as well as the channel H1:PSL-PWR_NPRO_OUT_DQ that is also connected to this PD; these now read correctly at ~1.86 W.

We left the NPRO running overnight, so there will be more data collection for looking for glitches, although we also did not see any while working in the PSL enclosure this afternoon.  The enclosure environmental controls are also running, but the lights are off.

On deck for tomorrow:

• Tweaking the board in the NPRO power supply that provides the NPRO readbacks for PSL Beckhoff
• Find and implement a mode matching solution for NPRO to Amp1
  • This is likely to be a 3 lens solution, with 1 lens getting the beam the correct size for the PSL EOM (new L01), and 2 lenses between FI01 and M03 that mode match to Amp1
• Begin recovering the amplifiers

TITLE: 10/22 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Tony (cancelled for maintenance)
SHIFT SUMMARY:

IFO is in NLN and MAINTENANCE. IFO will be DOWN for coming few days (estimated 2-3).

Extremely busy 8 hour maintenance day. The main events were the start of earth moving for the new LHO building, the PSL laser NPRO swap and the CO2Y Laser Swap. Details on all maintenance activities below.

• Earth moving by staging building (FAC)
  • Alog: None, Work Ongoing
  • Work Permit: 12120
  • Status: A lot of excavating and watering done today. A platform can be seen dug out behind the staging building. Ongoing for next few days.
• Guardian reboot (CDS)
  • Alog: 80825, 80818
  • Work Permit: 12145
  • Status: Initially had issues but ultimately successful.
• CO2Y Laser Swap (TCS)
  • Alog: 80812
  • Work Permit: 12142
  • Status: Done
• SQZ beam profile of beam pump (SQZ)
  • Alog: None
  • Work Permit: 12115
  • Status: Done, dicovered that the hold on the AOM box were too small.
• MX, EX Turbopump maintenance (VAC)
  • Alog: None
  • Work Permit: 12137
  • Status: Done
• MY EY quarterly turbopump functionality tests (VAC)
  • Alog: 80819
  • Work Permit: 12147
  • Status: Done
• Safety Committee walkabout (SAF)
  • Alog: None
  • Work Permit: None
  • Status: Done - committee went to all out-stations and cornerstations along with office areas.
• 3IFO Audit - manlift in West Bay (3IFO)
  • Alog: None
  • Work Permit: 12144
  • Status: Done.
• CR - Biergarten SEI Guralp sensor huddle tests (EE)
  • Alog: None, Work Ongoing
  • Work Permit: 12139
  • Status: Ongoing
• HAM2 OLG script test (SEI)
  • Alog: None, Work Ongoing
  • Work Permit: None
  • Start:17:20 UTC End: Ongoing (not today)
  • Status: Ryan C, Jim and Erik worked on this in the control room for a few hours. Switched from HAM5 to HAM2 for noise. Seemingly ongoing.
• FCES Clean air compressor run (VAC)
  • Alog: 80823
  • Work Permit: 12148
  • Status: Done
• Quarterly grease and rotation of VEA AHU supply fans (FAC)
  • Alog: None
  • Work Permit: 12149
  • Status: Done
• DC Power supply replacement for EY CNS-II GPS Unit (CDS)
  • Alog: 80825, 80817
  • Work Permit: 12150
  • Status: Done
• Snap file difference resolution (CDS)
  • Alog: 80825
  • Work Permit: 12151
  • Start: Done
• ALS Cable work on X/Y PDH (ALS)
  • Alog: 80826
  • Work Permit: None
  • Status: Done
• NPRO Work (PSL)
  • Alog: 80824, 80815,
  • Work Permit: 12153, 12155
  • Status: Jason and Ryan S (with support from Jenne) installed a new NPRO as planned and then watched to see if there were any glitches. If there were, they would know that it wasn't the NPRO as the culprit. Seeing as there were not, they made the decision to swap out the old NPRO entirely. This will take H1 down for 3-4 days as they bring the PSL back up (time for course alignment, mode matching, etc.). This is still ongoing.
• Disassembly/removal of old fume hood in the OSB Bake lab followed by any prep required for this space to be used differently (FAC)
  • Alog: None
  • Work Permit: 12119
  • Status: Done

LOG:

TJ, Camilla, Fil. 

WP12142,  Table layout: T1200007. Swapping because current CO2Y laser was at the end of it's gas life and slowly deteriorating, details in 79560. Prep done in 80679.

• Turned off: CO2 chassis power supply, the chiller and power supply in mechanical room mezzanine, AA chassis in the CER.
• Disconnected table water lines (outside CO2 table). Drained table lines with gravity and wet-vac.
• Disconnected all cables and water pipes to CO2X/RF driver and removed laser and RF driver. 
• Installed new CO2 laser
• Installed new RF driver after Fil reattached it's electrical plug (photo attached of connections)
• Reconnected all cables and water pipes, photo.
• Powered on CO2Y chiller in the mechanical room, while TJ was at the table to watch for leaks. 
• After a lunch break we checked for small leaks and needed to re-seat one of the copper pipe to RF driver 1/4" pipe fittings (note that we should buy more of these P/N 825-00001-000) did this with chiller off and table disconnected form main chiller line.
• Turned the laser on and measured ~49W out of the head with the S322C power meter. I updated the "Watts" filter in TCS-ITMY_CO2_LSRPWR_HD_PD from a gain of 9.12 to 12.2 so that the channel is correctly calibrated.
•  Turned on laser in PWM mode for low power realigning. We touched M1 and PM1 to try and improve alignment though the first two irises and used PM1 and PM2 to align to the last two irises.
• Checked centering on the FLIR, looked the same as last week 80679, photos with annular, central and no mask attached. 
• Checked beam isn't clipping any optics and is getting to the power meter after the waveplate. FLIR turned off. 
• Trend of laser power shows that its settling around 48.5W. 

Still to do:

• Tidy up the laser, wet-vac and bucket that are drying in the LVEA
• Calibrate the waveplate power control 
• During commissioning time: spend ~2 hours realigning in-vac using HWS and periscope picos. 

WP12150 Replace EY DNS-II GPS receiver power supply

Erik, Fil:

The power supply for the CNS-II GPS receiver was replaced. Erik found that the unit needs to be unplugged for at least 30 seconds or it doesn't initialize correctly.

WP12123 Install python3 alarms system as production

Dave:

The new python3 alarm code, which also uses Twilio for cell texts, was installed as the production code on cdslogin.

WP12145 h1guardian1 reboot

Erik, Jonathan, TJ:

h1guardian1 was rebooted to test the new boot drive which was installed recently. Please see Erik's alog for more details.

WP12151 Correcting missing channels in SDF snap files

Dave:

h1susitmpi's OBSERVE.snap was missing 4 channels. Since its settings were identical to those in safe.snap, OBSERVE.snap was changed to be an identical symlink as safe.snap

lrwxrwxrwx 1 controls advligorts 65 Oct 22 10:11 OBSERVE.snap -> /opt/rtcds/userapps/release/sus/h1/burtfiles/h1susitmpi_safe.snap
lrwxrwxrwx 1 controls advligorts 65 May  2  2016 safe.snap -> /opt/rtcds/userapps/release/sus/h1/burtfiles/h1susitmpi_safe.snap
 

The HEPI models h1hpietmy, h1hpiham4, h1hpiham5 and h1hpiham6 had safe.snap files were all missing the same 9 channels (see attachment).

These channels were added to the safe.snap files. To test, each models' SDF was transitioned from OBSERVE to safe and back to OBSERVE.

1. Temporary AO/AI connection to end station PDH CM board

Since ALS locking could be a time sink at times, and since going to the end station each time we want to make PDH measurement is a pain, I made the following changes.

CM boards:

Note that A=1000 sinusoidal excitation in AO results in 115mV RMS (measured at EY).

2. EY: Connecting DC output of the green PDH diode to H1:ALS-Y_REFL_B_LF

H1:ALS-Y_REFL_B_LF, which is supposed to be recording the first PD on Matt's PD interface box inside the ISCTEY, has been recording noise. I connected the DC output of the green PDH diode to the first channel on Matt box with the gain of x1.

(Before this change, the DC output of the green PDH diode was found to be routed to the "TEST" TNC feedthrough inside the enclosure on the same feedthgough panel as the RF output and DB15. There was no connection from "TEST" to anywhere outside the table enclosure.

Also, the first channel of Matt box has a cable called "ISCTEY-ALS-PD6" together with a cable for DC supply, but the other end of that cable was just dangling in air over the NPRO.

I disconnected the PDH diode DC output from the feedthrough panel and rerouted it to the ALS-PD6 cable that is still dangling over the NPRO. Since I didn't have a BNC barrel I used a T. Not too proud but it should be working, see the picture.)

3. EX: Could not find where the DC output of the green PDH diode cable goes.

Like EX, H1:ALS-X_REFL_B_LF have been recording noise. The first channel of Matt box doesn't have any cable connected.

However, unlike EY, the DC output cable of the green PDH diode seems to go to somewhere inside the sea of cables on the cable tray in the direction opposite to the "TEST" feedthrough. The cable was tied to the tray using multiple cable ties together with other cables rather tightly and I couldn't find where the other end was. I gave up as I wasn't ready to undo all cable ties on the tray.

[Writing on behalf of Jason and RyanS, who are actively working on the NPRO swap]

TL;DR: We're beginning the process of swapping for a spare NPRO.

This morning, Jason and Ryan went into the PSL enclosure and mounted a spare NPRO (they will remind me of which serial number later) on the PSL table, and cabled it up.  They moved the PD that serves H1:PSL-PWR_NPRO_OUT_DQ over near that spare NPRO and used a spare pickoff mirror to direct a small amount of power from the spare NPRO to the PD.  This test was to see if the spare NPRO exhibits glitches in the same way that the original NPRO does.  We had decided yesterday at the commissioning meeting that if the spare NPRO, after having sat for ~2 hours, does not exhibit glitching behavior, that we would go ahead with the swap.

In the attachments, blue ndscopes are of the original NPRO, that has been in service for all of O4 to-date.  The pink ndscopes are the spare NPRO, in this test setup.

With the original NPRO, you can see a signal when the door to the laser room opens and closes, and also you can see some glitches that are the same order of magnitude as the door.  This gives me some confidence that if we do see the door with the spare NPRO, then we should be able to see glitches if they are there.  With the spare NPRO and pickoff, the signal on the PD is much lower.  However, you can still see when the door opens and closes.  We let the laser sit for about 100 minutes, and didn't see any obvious glitching. 

• Attachment 1: Orig NPRO, in main laser path, you can see a DC change when the door opens/closes, and you can also see glitches.
• Attachment 2: Orig NPRO, in main laser path.  Same y-axes limits as Attach1.  Orig NPRO sometimes is weird and fuzzy.  Fuzz is almost same size as glitches.
• Attachment 3: Spare NPRO, in test path, but with otherwise all same electronics, same readout PD (different pickoff mirror, so lower signal). You can see a small change when the door opens/closes.
• Attachment 4: Spare NPRO, in test path. Same y-axes limits as Attach3. There is some drift due to (I think) temp change in the PSL enclosure, but no obvious glitches.

Since this spare NPRO is not obviously exhibiting glitching behavior, they are beginning the process of putting in that spare NPRO.

We anticipate that this work will take a few days.  We're doing it now since we can't lock during the daytime (due to excess ground motion this week), and LLO is offline for another few days pumping.

J. Kissel
WP 12140

I've completed 6 SUS + 4 ISI = 10 of 12 total DOF excitations that I wanted to drive before I ran out of time this morning. Each drive was "successful" in that I was able to get plenty of coherence between the 4 DOFs of ISI drive and SUS response, and some coherence between 6 SUS drive DOFs and ISI response. As expected, the bulk of the time was spent tuning the ISI excitations. I might have time to "finish" the data set and get the last two missing DOFs, but I was at least able to get both directions of LPY to LPY transfer functions, which are definitely juicy enough to get the analysis team started.

Measurement environmental/configuration differences of the HAM2 ISI from how they are nominally in observing:
    - PR3 M1 DAMP local damping loop gains are at -0.2, where they are nominally at -1.0. (The point of the test.)
    - CPS DIFF is OFF. (needed to do so for maintenance day)
    - Coil Driver z:p = 1:10 Hz analog low-pass (and digital compensation for it) is OFF. (need to do so to get good SNR on SUS M1 drive without saturating the SUS DACs)

Interesting things to call out that are the same as observing:
    - The PR3 alignment sliders were ON. P = -122 [urad]; Y = 100 [urad]. (Don't *expect* dynamics to change with ON vs. OFF, but we have seen diagonal response change if close an EQ stop. Haven't ever looked, but I wouldn't be surprised of off-diagonal responses change. Also DAC range gets consumed by DC alignment request, which is important for driving transfer functions.)
    - Corner station sensor correction, informed by the Bier Garten "ITMY" T240 on the ground. (the h1oaf0 computer got booted this morning, so we had to re-request the SEI_CS configuration guardian to be in WINDY. The SEI_ENV guardian had been set to LIGHT_MAINTENANCE.)
    - PR3 is NOT under any type of ISC global control; neither L, P, or Y. (global ISC feedback for the PRC's LPY DOFs goes to PRM and PR2.)

There are too many interesting transfer functions to attach, or even to export in the limited amount of time I have. 
So -- I leave it to the LSC team that inspired this test to look at the data, and use as needed.

The data have been committed to the SVN here:
    /ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/PR3/SAGM1/Data/
        2024-10-15_1627_H1SUSPR3_M1_WhiteNoise_L_0p02to50Hz.xml
        2024-10-15_1627_H1SUSPR3_M1_WhiteNoise_T_0p02to50Hz.xml
        2024-10-15_1627_H1SUSPR3_M1_WhiteNoise_V_0p02to50Hz.xml
        2024-10-15_1627_H1SUSPR3_M1_WhiteNoise_R_0p02to50Hz.xml
        2024-10-15_1627_H1SUSPR3_M1_WhiteNoise_P_0p02to50Hz.xml
        2024-10-15_1627_H1SUSPR3_M1_WhiteNoise_Y_0p02to50Hz.xml

    /ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/PR3/Common/Data
        2024-10-15_1627_H1ISIHAM2_ST1_WhiteNoise_PR3SusPoint_L_0p02to50Hz.xml
        2024-10-15_1627_H1ISIHAM2_ST1_WhiteNoise_PR3SusPoint_T_0p02to50Hz.xml
            [ran out of time for V]
            [ran out of time for R]
        2024-10-15_1627_H1ISIHAM2_ST1_WhiteNoise_PR3SusPoint_P_0p02to50Hz.xml
        2024-10-15_1627_H1ISIHAM2_ST1_WhiteNoise_PR3SusPoint_Y_0p02to50Hz.xml


For the SUS drives templates, I gathered:
     Typical:
     - The top mass, M1, OSEM sensors, in the LTVRPY Euler Basis, calibrated into microns or microradians, [um] or [urad].
         H1:SUS-PR3_M1_DAMP_?_IN1_DQ             [Filtered with the 64x filter, then downsampled to to fs = 256 Hz]
     - The top mass, M1, OSEM sensors, in the T1T2T3LFRTSDD OSEM Sensor/Coil Basis, calibrated into microns, [um].
         H1:SUS-PR3_M1_OSEMINF_??_OUT_DQ         [Filtered with the 64x filter, then downsampled to to fs = 256 Hz]
     - The top mass, M1, OSEM coils' requested drive, in the T1T2T3LFRTSD OSEM Sensor/Coil Basis, in raw (18 bit) DAC counts, [ct_M1SUS18bitDAC].
         H1:SUS-PR3_M1_MASTER_OUT_??_DQ          [Filtered with the 32x filter, then downsampled to to fs = 512 Hz]

     For this set of templates:
     - The bottom mass i.e. optic, M3, OSEM sensors, in the LPY Euler Basis, calibrated into microns or microradians, [um] or [urad].
         H1:SUS-PR3_M3_WIT_?_DQ                  [Filtered with the 64x filter, then downsampled to to fs = 256 Hz]
     - The bottom mass i.e. optic, M3, optical lever, in PIT YAW Euler Basis, calibrated into mircoradians, [urad].
         H1:SUS-PR3_M3_OPLEV_???_OUT_DQ          [Filtered with the 64x filter, then downsampled to to fs = 256 Hz]
     - The ISI's Stage 1 GS13 inertial sensors, projected to the PR3 suspension point LTVRPY Euler basis, calibrated into nanometers or nanoradians, [nm] or [nrad]
         H1:ISI-HAM2_SUSPOINT_PR3_EUL_?_DQ       [Filtered with the 4x filter, then downsampled to to fs = 1024 Hz]
     - The ISI's Stage 1 super sensors, in the ISI's Cartesian XYZRXRYRZ basis, calibrated into nanometers or nanoradians, [nm] or [nrad]
         H1:ISI-HAM2_ISO_*_IN1_DQ                [Filtered with the 2x filter, then downsampled to to fs = 2048 Hz]

Note: The six M1 OSEM sensors in the Euler Basis are set to be the "A" channels, such that you can reconstruct the transfer function between the M1 Euler Basis to all the other response channels in the physical units stated above. As usual the excitation channel for the given drive DOF (in each template, that's H1:SUS-MC3_M1_TEST_?_EXC) is automatically stored, but these channels are in goofy "Euler Basis (18-bit) DAC counts," so tough to turn into physical units.

For the brand new ISI drive templates, I gathered:
     - The ISI's Stage 1 super sensors, in the ISI's Cartesian XYZRXRYRZ basis, calibrated into nanometers or nanoradians, [nm] or [nrad]
         H1:ISI-HAM2_ISO_*_IN1_DQ                [Filtered with the 2x filter, then downsampled to to fs = 2048 Hz]
     - The ISI's Stage 1 GS13 inertial sensors, projected to the PR3 suspension point LTVRPY Euler basis, calibrated into nanometers or nanoradians, [nm] or [nrad]
         H1:ISI-HAM2_SUSPOINT_PR3_EUL_?_DQ       [Filtered with the 4x filter, then downsampled to to fs = 1024 Hz]

     - The top mass, M1, OSEM sensors, in the LTVRPY Euler Basis, calibrated into microns or microradians, [um] or [urad].
         H1:SUS-PR3_M1_DAMP_?_IN1_DQ             [Filtered with the 64x filter, then downsampled to to fs = 256 Hz]
     - The bottom mass i.e. optic, M3, OSEM sensors, in the LPY Euler Basis, calibrated into microns or microradians, [um] or [urad].
         H1:SUS-PR3_M3_WIT_?_DQ                  [Filtered with the 64x filter, then downsampled to to fs = 256 Hz]
     - The bottom mass i.e. optic, M3, optical lever, in PIT YAW Euler Basis, calibrated into mircoradians, [urad].
         H1:SUS-PR3_M3_OPLEV_???_OUT_DQ          [Filtered with the 64x filter, then downsampled to to fs = 256 Hz]
     - The ISI's Stage 1 actuators' requested drive, in the H1H2H3V1V2V3 ISI actuator basis, in raw (16-bit) DAC counts, [ct_ISIST116bitDAC].
         H1:ISI-HAM2_OUTF_??_OUT                 [Didn't realize in time that there are DQ channels H1:ISI-HAM2_MASTER_??_DRIVE_DQ stored at fs = 2048 Hz, or I would have used those.]

Note: Here, I set the number of "A" channels to twelve, such that both the ISI's Cartesian basis and the PR3 Suspoint basis versions of the GS13s can be used as the transfer function reference channel.