I reverted the Changes to ALS_ARM.py that were put in today because the DOF2 WFS were not working.

TITLE: 08/23 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 16mph Gusts, 11mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

Slowly going Locking states so commisioners can fine tune full IFO Y Arm alignment , by stepping through PREP_ASC_FORFULL_IFO[429] manually. This is to set the Y arm Green reference & document settings.

Naoki, Vicky, Camilla, Sheila. Copied instructions from 78115.

Today we couldn't find the SQZ beam while the IFO was locked using  SQZ-OMC_TRANS_RF3_DEMOD_RFMON as we'd been ale to do in 78088. When the IFO unlocked we used the SQZ SEED beam in a SQZ single bounce (much higher power so easier to see) to  improve the SQZ alignment. Expect when we're locked will still need to touch alignment, may get away with a few rounds of SQZ_MANAGER's SCAN_ALIGNMENT_FDS state (wiki notes). Notes below.

• Aligned SRM, misaligned SR2/3  (ITMs being used for ALS troubleshooting). FC misaligned.
• Increased seed from 1mW to 75mW
• SQZ Manager LOCKED SEED DITHER (need to revert the hysteritis offset Naoki made yesterday 79630... don't know why)
• HAM6/7 WD untripped
• First tried to align just to AS_C: kept centered on AS_C with ZM4 while moving ZM5 to increase power on AS_C.
• This didn't go well and Vicky/Naoki/Shiela suggested I should really care about AS_A/B.
• Aligned OM1,2,3,SRM to their alignments in today's full lock. Opened fast shutter.
• Then we walked, centering AS_C with ZM4 and AS_A with ZM6. Then we touched ZM5 to bring  AS_B closer to center and ZM6 to bring back AS_A . At this point we could also see the beam be un-clipped on AS AIR camera. We centered on this and mostly centered on the OMC QPDs.
• Similar/better powers than 78115:  Got peaks of 900e-6 on AS_C_SUM, 70 of AS_A and AS_B and beams were mostly centered (at least within 0.4). as AS_A/B/C, AS_AIR and OMC QPDs. Plot attached of end alignment.
• Accepted in ZM4/5/6 alignments in sdf and brought the seed power back to 0.8mW.

TITLE: 08/22 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Another busy day of IFO recovery. See Elenna's alog for a summary of the morning's activities.

Since then, H1 lost lock from a ringup of PI mode 24. Naoki, Camilla, and the SUS_PI Guardian all tried to damp it down to no avail; why do we now have a PI that we cannot damp?

After the lockloss, we started back with troubleshooting ALS issues while team SQZ worked on recovering their alignment. TJ and I tried to find an arm alignment that both had the arm power and NSUM counts on ALSY WFS_B high, but we weren't successful. Eventually, Keita was able to help us recenter on both WFS A and B by adjusting the biases for both PZTs on the ISCEY table (found on the secret ALS-Y_WFS Auto Centering medm screen) after we locked the arm in green. These new bias settings ahve been accepted in the ALSEY SAFE.snap SDF table (screenshot attached). It turns out that with the centering being so off on the ALSY WFS, the auto-centering servo was railing, which didn't allow the ALS WFS to engage without us forcing them to. After centering on the WFS, we took the ALS_YARM Guardian to LOCKED_SLOW_WFS_ETM_TMS and the ALSY WFS engaged properly.

The next steps will be to lock the full IFO keeping ALSY locked to set the green camera references.

LOG:

The transition from the piston driven (non-dried) instrument air compressor to the new (inline dried) scroll style compressor was finalized last week. The compressor is presently left at its factory set pressure of 120psi. TJ raised the high limit threshold for alarms to coincide with this change in pressure. The new system has been delivering the facility with air without issue since last week.

Amidst this transition, I elected to pressurize the compressed air line between the chiller yard and mechanical room and remove any feed/demands from it as noted in my previous alog. There were no significant pressure losses over an 8-hour period and as such I have no reason to believe that the line feeding the mechanical room is in any way compromised.

Not shown in the pictures: Insulation and heat trace on lines that require it and weatherproof outlet box

E. Otterman C. Soike T. Guidry

Just a quick report now that we are back to NLN!

• After getting through various alignment details at 2W, getting to NLN is very smooth and requires no intervention
• The problem with the PI guardian last night was because I manualed over the "inject squeezing" state which also engages the PI guardian. PI damping engaged properly today!
• Naoki is working on SQZ alignment, reverting the ZMs to the July alignment
• Some visible excess noise in DARM below 60 Hz. A quick coherence measurement shows that a portion of this noise at least is LSC. We can retune the FF soon
• I ran a quick broadband PCal measurement that shows our calibration is off, unsurprisingly. Louis can help us recover the calibration over the next few days.
• We have still not set the y-arm green references, due to issues getting the ALS Y locked properly when in fully alignment. We can fix this on the next relocking
• We still have not solved the ALS Y WFS problem, and we still don't know what's wrong. However, we have improved the counts on ALS Y WFS B, which might help DOF2 run better when the loops are engaged.
• The current (not-well-calibrated and unsqueezed) range is reading at 123 Mpc

h1cam26 (BS) stopped updating around 22:22 PDT Wed 21aug2024.

I power cycled the camera via sw-lvea-aux POE and restarted the process on h1digivideo2. 

h1cam26 is on switch port 0/35

The offload of the past 6 month's of raw minute trend files from h1daqtw1's SSD-RAID to permanent archival on spinning media is complete. TW1's SSD-RAID disk usage has been reduced from 92% to 2%.

First NDS1 restart: 09:56 Tue 20aug2024

Start file copy: 10:53 Tue 20aug2024

File copy completed: 12:07 Wed 21aug2024 (25hrs 14mins)

Second NDS1 restart: 15:47 Wed 21aug2024

Start file deletion on TW1: 09:32 Thu 22aug2024

File deletion complete: 11:40 Thu 22aug2024 (2hrs 8mins)

I wanted to check the last lock loss concerning the power upstream/downstream of the fast shutter, but quickly remembered that the upstream signal, i.e. the non-whitened analog version of the ASC-ASC_SUM, was disconnected from the fast PEM ADC (H1:PEM-CS_ADC_5_19_2K_OUT_DQ) before the last vent to fix OFI, and it was not reconnected since.

I went to the floor and restored the connection.

Original alogs about this connection: 78968 and 79077.

After HAM7/FC GVs were open this morning, I aligned FC1/2 and FC GR is locked. FC GR trans is ~110, which is reasonable. 

I measured NLG following 76542. The measured NLG is 16 (0.247/0.0154=16), which is a bit lower than the last measurement in 78000. The OPO temperature is very different from before as shown in the attached SDF.

Famis 26321 HVAC Vibrometer Check

HVAC fan channel  H0:VAC-EY_FAN1_ACC_INCHSEC saw a decrease in signal but an increase in noise on the signal starting on Tuesday.
H0:VAC-MR_FAN2_170_2_ACC_INCHSEC saw an increase in the counts signal starting on Tuesday.

Thu Aug 22 08:15:16 2024 INFO: Fill completed in 15min 12secs

TITLE: 08/22 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY: H1 is holding in PREP_FOR_LOCKING after losing lock last night (and was locked at NLN for almost 4 hours!). I'm starting to go through notes from locking attempts last night, and once I have, I'll start trying to bring up H1.

TITLE: 08/21 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

LOG:

• 2320-2338 Going out to center OMC trans camera (tj, camilla)
• 1650 Started Initial Alignment & then went into locking
• 0407 PI Mode 24 alerts came up.  I went to the PI Overview screen, found the SUS PI guardian, and saw it was in the IFO DOWN state!  So I transitioned it to its nominal state of PI Damping.  But looks like PI24 is railing around 2000 on nuc25.

[Ryan S, Corey, Camilla, TJ, Elenna]

Quick, must be fixed ASAP summary: something is wrong with ALS Y. The WFS are not engaging when they should. DOF2 seems to be bad right now, and the camera offsets are also not good. Guardian does not notice the WFS are not engaging. Do not trust the green camera offsets on the Y arm.

Ryan S brought the IFO to DC readout successfully, and began damping violin modes. Some noisy peaks are evident, probably due to pumps that are still running and/or the HAM5/7 relay tube gate valve is still closed.

I set SRC1 P and Y offsets and engaged the loops. SRC1 P offset = -0.538 and SRC1 Y offset = 1.327

These offsets keep us in a good place for the SRM, and then I can walk SR2 to converge the SRC2 loop.

This worked for pitch, and I was just about to engage SRC2 pitch when there was a lockloss due to HAM6 motion. Both the vacuum team and Camilla and TJ were around that area for various activities, potentially causing the lockloss.

After TJ and Camilla aligned the OMC trans camera, we ran initial alignment (trying to lock DRMI failed because the alignment was very poor).

Initial alignment failure:

• we tried to run a full initial alignment with the new green camera offsets, but it failed
• ALS locks successfully for both arms, and the X arm completes "initial alignment offloaded"
• However, the Y arm gets stuck in the "initial alignment" state and the WFS do not turn on
• We found a button that can "force" turn them on, but the alignment pulls the Y arm in a bad direction
• We tried this a few times, turning off various alignment loops to pinpoint which ones were bad
• it appears that dof2 and the camera loops are not good
• I don't know if there is some other issue I am missing, but it doesn't seem like the camera offsets are in the right place (even though they should be from earlier!)
• We couldn't figure out why the ALS guardian would just hang in "initial alignment" either, waiting for the WFS to converge when it wasn't even engaging the WFS. Ryan hypothesized it was a "beckhoff issue"

I had us give up and run a manual initial alignment without using the green cameras.

It appears that even in "Locked slow wfs" the ALS Y WFS do not turn on, but it's somehow not a problem in the guardian because the error signals are pretty well converged already? Something is going wrong with the ALS Y WFS. Trending back it seems that the ALS Y WFS did not run at all today, and we didn't seem to notice!

In order to properly run the WFS, I have to take the ALS Y guardian to "locked slow ETM TMS WFS" and then "force" the autocentering trigger in order to turn on the WFS. Then I can let the guardian move on to offloading.

Additionally, ALS Y DOF 2 is taking the Y arm in a bad direction again after it converges the alignment in green locking. Once I force on the Y arm WFS, I turn off the DOF2 signals and that keeps the Y arm stable through the next locking steps.

To get the SRC ASC on:

• Corey and I went to "prep asc for full ifo"
• I stepped through by hand, skipping the SRC ASC steps
• I set offsets in SRC1 at about -0.5 for pitch and 1.4 for yaw.
• I engaged the SRC1 loops successfully with the offsets on
• I stepped SR2 in pitch and yaw to converge the SRC2 error signal and keep the buildups good
• I engaged SRC2
• I turned on soft loops, corner alignment got worse (POP18 dropped, POP90 increased)
• to improve the POP buildups, I then slowly stepped the SRC1 offsets to zero, which improved the alignment (and SR2 and other optics could follow!)
• These offsets are SDFed for now, but should be turned off once we know the alignment is "saved" properly otherwise

Indeed now that we are in full good alignment with ASC engaged, the ALS Y green is no longer locked on a 00 mode, which means that the camera offsets from earlier are maybe incorrect. However, I could not get the ALS Y to lock on the 00 beam in order to reset the offsets. First, I tried moving the green QPD offset sliders around, then I tried very slowly moving the TMS in 0.1 urad steps. All I can get on the camera is this.

However, I did another set of the ALS X green references (very minor change), with the SDF screenshot attached. The ITMX CAM values were still good, so I didn't change them.

Everything else executed well: power up, low noise transitions. I skipped over "inject squeezing" since we don't have the valves open for the FC yet. We have arrived in OMC whitening, where the guardian is holding us to damp the violins.

DARM is very noisy, probably due to whatever vacuum pumps are still running and any extra noise due to the relay tube being closed. Most calibration lines have no SNR due to the high noise.

Major problem to solve is the ALS Y alignment and camera offsets, but locking has succeeded and we are almost back to nominal!

Today's activities:

- The large LVEA turbos (OMC, X-manifold, Y-manifold) have been valved out
- The large Gate Valves (GV5 and GV7) have been opened
- The large turbos have been switched off - see their statistics in the comments
- The HAM6 RGA bakeout is being ramped down. Tomorrow it should be ready to be valved in, and make the 1st scan.

Tomorrow the whole FCT will be valved together with the corner, so everything will go back to normal. The reason why it will only be done tomorrow is that the pressures of the corner need some time to catch up with HAM7

The pressures:
Corner (PT-120 and PT131): 7.5E-8 Torr
HAM6: 2.5E-7 Torr
HAM7: 7.45E-8 Torr
CP1: ~3E-8 Torr
CP2: ~5E-8 Torr

TITLE: 08/21 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

Arm gate valves (GV5 and GV7) were opened this morning, which allowed us to start recovering the IFO. See Elenna's alog for details in the process. Currently running through a full initial alignment now that the green camera references have been set.

Other items to note from the day are that the LVEA remains laser HAZARD following the camera adjustments on HAM6.

LOG: