Commissioning plans take us to ~1600 PT (2300 UTC). Brief list of plans (not necessarily in order):
Camilla, Jonathan, Erik, Dave:
Erik went to EY and took a look at the h1hwsey computer. It had a stack trace display on its console which does not appear to give any clear indications of the crash.
Subsequent computer reboots and power cycling (including power cord pull out) would not boot the computer. The FAIL red LED is on, the power green led is off, nothing appears on the local console.
Camilla says we can run for the rest of the week without ETMY HWS.
I have green'ed up the EDC on the CDS Overview, where GREEN=86 disconnected channels.
Since last weekend ITMY mode05/06 were rung up and the nominal settings were not able to damp them. During that time the microseism and ground motion was higher than nominal, also LVEA temperature was slightly elevated. On Monday I found a new setting which worked all day and during evening shift (as confirmed by Ryan C). However it stopped working this morning. Hence, I again tried the nominal settings and it seems to be working for now. Given below are the details,
Settings working today (Nov 01 2023): FM6 + FM8 + FM10 Gain -0.01 (will slightly ramp it up later, although we lost lock while I was writing this alog, nevertheless I will try again) - see plot attached.
Settings which seems to be working on Monday (Oct 30 2023): FM6 + FM10 Gain 0.01
I will keep a watch on them before committing it in the lscparams.
This afternoon during the 2nd lock of the day I've had continued success damping ITMY mode5/6 with FM6+FM8+FM10, G= -0.025 (I upped the gain in 3 steps).
It may have been the earthquake that rang up the violins but there was a large gap in the DCPD data from us being down. But during a time that we were in IA the DCPD signals were larger (by almost a factor of 2, DCPD Min/Max difference of 25k vs 40k) than what they usually are.
Wed Nov 01 10:13:20 2023 INFO: Fill completed in 13min 16secs
Jordan confirmed a good fill curbside.
We were only locked for 1:08, lockloss at 19:29UTC. ITMX was the first suspension to saturate right before the LL and there was a 500Hz oscillation in LSC-DARM which was probably the violin modes.
Recovered NLN at 18:22uTC, back into observing from 18:42 till 19:14
I'm beginning to think that this may have been caused by the SOFT Yaw loops. The quads see some type of deflection in their master out before the LSC DARM channel sees anything, and I tracked it down to SOFT Y so far. I'll do some more digging on this tomorrow and see if I can keep following this thread.
We recovered NLN at 16:10UTC after damping violins for about a half hour and Observing at 16:46UTC, VIcky touched up the SQZer after it was having issues and Erik went out to investigate the EY HWS computer issue but was not able to resolve it as it seems dead. We have comissioning planned for 12pm - 4pm PST, 19:00 - 23:00 UTC. GDS was also down but has been brought back.
Jordan, Gerardo
Yesterday during Tuesday maintenance (10/31/23), we were able to install the final two of the new 150 l/s ion pumps on FC-B section, B4 and B5 crosses. The Ion Pump/Tee/Angle valve assembly was pre-built in the staging building and then pumped down and leak checked. This assembly was stored under vacuum and then brought to the LVEA.
We closed FCV-3 (BSC3), FCV-4, FCV-5, and FCV-6 (HAM8) to isolate the B4 and B5 crosses. Then closed the -Z axis GV on the two crosses to isolate the ion pump port. We then vented the ion pump assembly with N2 and removed the angle valve/6" zero-length reducer on the cross, and installed the ion pump on the 6" CF port. A genie lift was used to lift/hold the ion pump while the connections were made.
Once installed, we used the leak detector/small turbo to pump down the assembly, and then helium leak tested the one CF connection that was made. There was no detectable signal above the helium background of 9.5E-11 Torr-l/s.
The ion pump was powered on locally, and quickly dropped to ~2mA. The -Z gate valve remains closed, and the rest of the FCT gate valves were reopened once the ion pump was leak checked and powered on. This concludes the installation of the ion pumps, high voltage conduit and communication cables to follow in the following weeks.
TITLE: 11/01 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 2mph Gusts, 0mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.28 μm/s
QUICK SUMMARY:
The violins are looking pretty rung up on DARM as we're in ENGAGE_ASC_FOR_FULL_IFO
We're waiting in OMC_WHITENING damping violins and will be for maybe a half hour? Hopefully less
We've reaquired NLN at 16:10UTC but the SQZ FC is struggling to lock.
FC was quite misaligned and not able to find a TEM00 mode for locking; I had to walk FC2 quite far to find resonance, and I also touched ZM3, see slider screenshots accepted into SDF here: ZM3 / FC2 slider diffs to relock. I probably didn't need to move ZM3. Not sure how this will fit into our recent SQZ ASC issues 73855.
Other than sliders, accepted the folllowing FDS SQZ SDFs in Observe (after LHO:73876 accepted SDF's in safe). Includes SQZ+FC ASC ON switches, and FC + LO servo locking settings.
I tweaked up sqz angle and opo temp ; in particular sqz angle seemed off by 10+ degrees even if opo temp was the same. The sqz angle may need to be adjusted later today in commissioning, since I set it for squeezing pretty early on (~30min) into lock, so expect IFO was unthermalized.
TITLE: 11/01 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
Inherited H1 during Initial Alignment, stuck on ACQUIRE_XARM_IR.
Tried locking the IFO just to see where H1 will stumble.
Tried Initial Alignment and got stuck at Acquire_XARM_IR.
Tried Man Alignment and got stuck in the same spot.
Found my own alog about a very similar situation:
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=69497
Started reading Troubleshooting Docs-
Checked TRX OPD , Trending PR2.
reverting PR2 to GPS time : 1382733517
reverted xarm to gps time 1382733517
Retouched up ALS x arm in green arms manual tried to do an initial alignment again and got stuck at
ACUIRE_XARM_IR again
~ 9:47 UTC Reverted the PRC PRM, PR1,PR2,PR3 to 1382733517
Success!! INITIAL ALIGNMENT moved forward!!!
INITIAL ALIGNMENT completed!
Locking Got Stuck at ALS and Find IR. So running Another initial Alignment. Perhaps moving PR Cavity sliders half way though Initial alignment leads to a bad Alignment, with ALS? Perhaps my movement of PR3 was too much?
Still Cant get past Find IR After another IA.
Jenne had popped in to Matter most and Given me a clue to take notes on where PR3 was.
and PR3 and what I was setting it to.
Notes:
PR3P After: -117.4
PR3Y After: 155.3
PR3P before: -116.2
PR3Y Before : 152.1
I'm exausted and spinning my wheels. I think this might still be a PR Cavity issue. But I can no longer think clearly and feel like I'd make irratic changes at this time in the morning. I've gotta step away and get some sleep.
LOG:
No log
Apologies for the scattered notes; I'm on a meeting working on the IFO in parallel.
ISC_LOCK reset - including prep_for_locking and sdf_revert.
Locked and aligned green arms.
Set XarmIR part of Align_IFO to Input_align, so that when I move PR3 then PR2 and IM4 will have to follow along. Moved PR3 in the direction of where the oplevs were ~14 days ago, although didn't go all the way since the COMM beatnote went past a peak and started getting less good. Had ALS_XARM at Locked_slow_no_wfs, so could see that the comm beatnote improved from -16dB to about -10dB. (I could move PR3 to get up to about -8dB, but that isn't back in the direction of where it had been 2 weeks ago).
Offloaded Input_align.
Going through the rest of initial alignment . PRC align went without intervention, MICH bright align went without intervention. SRC needed help, so I asked it to do SR2_align. SRC still wouldn't catch, so I touched SRM in yaw and it caught and finished initial alignment.
Locking:
ISC_LOCK automatically went through PRMI_ASC. Automatically went back to DRMI and caught lock after that. Locking going smoothly so far, so handing back to daytime operator.
Chiller 1 had tripped its main breaker due to a bad condenser motor, which also took out a 30 amp fuse. Bad motor will be replaced when a replacement is available. Chiller 2 is currently in lead position.
attached trends since the chiller swap (courtesy of D. Barker):
Jenne notes that this changing high frequency noise during the first hours of a lock since Wed 25th Oct (purple trace in 73798) may be caused by the new higher CARM gain 73738, changed on that day.
In 73798, I noted The 4.8kHz noise that suddenly changes ~ 1h40 into NLN, Jenne suggests this could be aliased down CARM gain peaking. Looking at the 64kHz channel, plot attached, noise disappears from 18.4 to 18.7kHz (v. large peak) and appears at 16.6kHz to 16.8kHz and at 21.1kHz.
We have had a weekend of shorter (~5hour) locks and two locklosses form LASER_NOISE_SUPPRESSION state#575 (73787, 73831), the state this CARM gain is changed. Maybe this gain change has made us less stable, we'll discuss today reverting it.
CARM sliders reverted back to 6dB in ISC_LOCK (svn) and loaded.
On Monday, Naoki and Sheila 73855 saw that even with the CARM gain back at 12dB, the high frequency squeezing was still bad and the optimal H1:SQZ-CLF_REFL_RF6_PHASE_PHASEDEG sqz angle has to be adjusted a lot.
Maybe the CARM gain increase was effecting stability, but we don't think it wasn't causing the high frequency noise that was present in FDS, FIS but not no SQZ, plot attached. With adjustment to SQZ angle the SQZ greatly improved. It wasn't clear to us why the SQZ angle changed.
Although the overall high frequency noise was still bad once the CARM gain was reduced from 12 back down to 6, the peak around 4600Hz did disappear once the CARM gain was reverted. See attached SQZ BLRM 6 purple trace with CARM at 12 and CARM at nominal 6.
See attached high frequency plot showing peaks at 16.4kHz and 18.7kHz (purple) and thermalized around 16.6kHz (red), peaks disappeared once CARM gain was reduced ( green to blue traces). Maybe this concludes that the peaks are CARM gain peaking as Jenne suggested.
TITLE: 05/11 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Aligning
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 6mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.25 μm/s
QUICK SUMMARY:
IFO'S Current Status: unlocked and misaligned.
Hear we go:
21 seconds into my shift there was a lockloss
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1367794839
ITMY Mode 8 Violin is ringing up again. Changed the gain from -0.4 to 0 again, and I'll be watching it.
00:32 UTC Lockloss for an unknown reason
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1367800386
Dolphin error cause a lockloss and the SUS models needed to be restarted.
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1367803504
Everything on OPS Overview was Red and all the watch dogs had been tripped.
I Reset all suspension WDs, took all suspensions to safe, then reset all ISI's WDS, then Reset ALL HEPI WDs.
Finally got every thing back into what I thought was working order but relized that I don't have any light on the ALSX and ALSY cams.
I am now running the Dither scripts noted in alog 62024 for both X and Y arm.
AS Air looks oddly center on the AS Air cam instead of it's natrual state of top right.
The dither scripts took AS air cam back to what I would consider it's natural state in the top left.
I now have light on both arms! Yay!
Me and Increase flashes became best of buds while we both tryed to get X arm above 0.50.
ETMX TMSX ITMX PRM and PRM3 all got moved whle trying to increase flashes.
I wasted a hour an hour trying to find information on adjusting the OPLEVS and never ended up finding them let alone touching them.
I eventually manage to get a good ALS X & Y alignment after time machining back to look at a number of sliders on the IFO_ALign screen and re-adjusting PRM and PR3.
Started an INIT_ALIGN and got stuck in ACQUIRE_XARM_IR, Then spend an hour or 2 searching through the troubleshooting Docs & alogs to try and find a way to get past this:
2023-05-11_07:33:31.707994Z ALIGN_IFO [ACQUIRE_XARM_IR.main] ezca: H1:LSC-XARM => ON: INPUT
2023-05-11_07:33:31.709570Z ALIGN_IFO [ACQUIRE_XARM_IR.main] timer['pause'] = 3
2023-05-11_07:33:31.785294Z ALIGN_IFO [ACQUIRE_XARM_IR.run] timer['pause'] = 0.5
I then tried to cancel out of INIT_ALIGN because I remember I heard there were some problems with getting through INIT_ALIGN a few days ago. Then my plan was to try and have ISC_LOCK's FIND_IR stage in the Locking sequece get me past that spot in the Alignment.
Would you be Surprised if I said that IR was not found? I had to touch both up COMM and DIFF.
I then got stuck in Check Mitch Fringes. And bumped the BS around until it got stuck in Aquire PRMI Again. The Troubleshooting document mentions MICH_Dark_LOCK but not Mitch fringes.
I then Spend an hour or 2 looking for information on that.
I then tried to restore ALL Sliders back to 24 hours ago when we were locked.
and tried the above process over again.
Then I tried restoring ALL Sliders back to a time where we were unlocked but had Good ASL X&Y flashes right before a lock and tried the process over again.
I believe I am missing some important nuggets of knowlege on how to get IFO locked from a misaligned state.
Things I wanted to do But couldn't figure out how: Check OSEMS and OPLEVS.
Potential Solution to this because I found myself in this situation again last night see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=73891 with Jenne's notes.
If you find your self stuck in this postition again, specifically when PR3 has been drifting due to temperature changes or something.
Note the COMM beat note, this may be a clue as to what is going wrong. If the magnitude of the COMM beat note is too low below ~ -16 , Follow the instructions in Jenne's comment on 73891 linked above.
OR
First: Go back to green arms locked and touch PR3 while watching the Comm Beat note aiming for (-8 to -4) depending on conditions. Beawre that the Diff beatnote may change for worse and we must balance this with the optimization of the comm beat note.
The go back to initial alignment. And if that doesn't work then follow Jenne's comment on 73891 .
Good luck
