TITLE: 05/11 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Aligning
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 7mph Gusts, 6mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.25 μm/s
QUICK SUMMARY:
IFO'S Current Status: unlocked and misaligned.
 

Hear we go: 
21 seconds into my shift there was a lockloss
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1367794839

ITMY Mode 8 Violin is ringing up again. Changed the gain from -0.4 to 0 again, and I'll be watching it.
 
00:32 UTC Lockloss for an unknown reason
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1367800386

Dolphin error cause a lockloss and the SUS models needed to be restarted. 
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1367803504

Everything on OPS Overview was Red and all the watch dogs had been tripped.
I Reset all suspension WDs, took all suspensions to safe, then reset all ISI's WDS, then Reset ALL HEPI WDs.

Finally got every thing back into what I thought was working order but relized that I don't have any light on the ALSX and ALSY cams. 
I am now running the Dither scripts noted in alog 62024 for both X and Y arm.  
AS Air looks oddly center on the AS Air cam instead of it's natrual state of top right.
The dither scripts took AS air cam back to what I would consider it's natural state in the top left.

I now have light on both arms! Yay!
Me and Increase flashes became best of buds while we both tryed to get X arm above 0.50. 

ETMX TMSX ITMX PRM and PRM3 all got moved whle trying to increase flashes.

I wasted a hour an hour trying to find information on adjusting the OPLEVS and never ended up finding them let alone touching them.  
I eventually manage to get a good ALS X & Y alignment after time machining back to look at a number of sliders on the IFO_ALign screen and re-adjusting PRM and PR3. 

Started an INIT_ALIGN and got stuck in ACQUIRE_XARM_IR, Then spend an hour or 2 searching through the troubleshooting Docs & alogs to try and find a way to get past this: 
2023-05-11_07:33:31.707994Z ALIGN_IFO [ACQUIRE_XARM_IR.main] ezca: H1:LSC-XARM => ON: INPUT
2023-05-11_07:33:31.709570Z ALIGN_IFO [ACQUIRE_XARM_IR.main] timer['pause'] = 3
2023-05-11_07:33:31.785294Z ALIGN_IFO [ACQUIRE_XARM_IR.run] timer['pause'] = 0.5

I then tried to cancel out of INIT_ALIGN because I remember I heard there were some problems with getting through INIT_ALIGN a few days ago. Then my plan was to try and have ISC_LOCK's FIND_IR stage in the Locking sequece get me past that spot in the Alignment.
Would you be Surprised if I said that IR was not found? I had to touch both up COMM and DIFF. 

I then got stuck in Check Mitch Fringes. And bumped the BS around until it got stuck in Aquire PRMI Again. The Troubleshooting document mentions MICH_Dark_LOCK but not Mitch fringes. 
I then Spend an hour or 2 looking for information on that.

I then tried to restore ALL Sliders back to 24 hours ago when we were locked.
and tried the above process over again.

Then I tried restoring ALL Sliders back to a time where we were unlocked but had Good ASL X&Y flashes right before a lock and tried the process over again.

I believe I am missing some important nuggets of knowlege on how to get IFO locked from a misaligned state.
Things I wanted to do But couldn't figure out how: Check OSEMS and OPLEVS. 

I was not on day shift for Tuesday maintenance, so Ryan Crouch made these measurments for me and sent me the files to post here.
FAMIS 25065

Current IFO Status : Increase flashes 
Dolphin error cause a lockloss and the SUS models needed to be restarted.
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1367803504

Spoke to Dave Barker about the CDS Dolphin errors and he got all the models restarted again for me.

I Reset all suspension WDs, took all suspensions to safe, Then reset all ISI's WDS, then Reset ALL HEPI WDs.

Finally got every thing back into what I thought was working order but relized that I don't have any light on the ALSX and ALSY cams.
I am now running the Dither scripts noted in alog 62024 for both X and Y arm.  
AS Air looks oddly center on the AS Air cam instead of it's natrual state of top right.
The dither scripts took AS air cam back to what i would consider it's natural state in the top left.

I now have light on both arms! Yay!
But I have been trying to get X-arm to increase it's flashes for a long while now and have been struggling to get flashes above 50%.
And so has the Increase flashes Guardian setting.
 

Tony, Dave:

At 18:24PDT we had a corner station Dolphin glitch which stopped the DACs on h1sus[b123, h2a, h34 and h56], see attached.

To recover, the IOP models (which means all the models) on these front ends were restarted.

Although the SUS DACs had been DACKILL'ed, the IPC continued to run and so the SWWD did not trip any SEI system.

Following the restarts I untripped the SUS SWWDs and handed the IFO over to Tony to complete the recovery.

Naoki, Camilla

See attached:

• at 2023/05/10 22:32:17 UTC to SQZ_MANAGER was in DOWN but the beam diverter was open
• at 2023/05/10 22:21:03 UTC SQZ_MANAGER was requested to NO_SQUEEZING but stayed at FDS.

We will investigate this logic tomorrow. We think we got into a strange state as the FC was struggling to lock in IR but Naoki double the IR gain and SQZ_FC is now working. 

TITLE: 05/10 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 6mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.19 μm/s
QUICK SUMMARY:
21 seconds into my shift there was a lockloss from NLN.
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1367794839

Temperatures in the LVEA seem to be climbing according to the Channels found on FOM 32
H0:FMC-CS_LVEA_ZONE1A_DEGF
H0:FMC-CS_LVEA_ZONE1B_DEGF
H0:FMC-CS_LVEA_ZONE4_DEGF
H0:FMC-CS_LVEA_ZONE5_DEGF
Picture of NDSCOPE at 23:00 UTC for reference is included.

IFO Current Status: Locking at AQUIRE_PRMI

Previously, I hacked together a high power DCPD counts checker in the OMC whitening state that worked well for 20 mA. We have changed the DARM offset, so it no longer worked well, and we have nearly saturated the DCPDs when turning on the OMC whitening during lock acquisition.

What I failed to understand when I first designed the checker, and thank Jenne for patiently explaining to me: the OMC whitening is frequency dependent, so any DC offsets (like the DARM offset) should be factored out of the calculation and added back in later, after the high frequency portion of the DCPD counts is multipled by 10.

Using the DCPD min/max scope, I eyeballed what count offset corresponds to DARM offset. It appears that at 40 mA, the offset is about 110,000 cts. This lines up with the eyeballed value at 20 mA DARM offset which is about 55,000 cts.

Now, the math in the "check_for_violins_saturation" (ISC_library function) at power=high goes like this:

• overall_max is estimated by taking the maximum of the recent 30 seconds of DCPD data
• Expected_counts = 10 * (overall_max - 100000) + 100000
• If this number is greater than 2^19, the function returns false, and we hold 60 seconds before checking again. If the function returns true, we turn on the OMC whitening and proceed to NLN.

Note: I chose 100,000 instead of 110,000 to overestimate what the counts would be with whitening on. This value is hard coded, and must be updated every time we change the DARM offset.

TITLE: 05/10 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
SHIFT SUMMARY: Large earthquake in Tonga kept us out for ~5 hours today, then a failed ADC card for IOPSEIH45. We made it up to low noise for 48 minutes and we just lost lock. Unsure as to why at the moment.
LOG:

• Lost lock at 16:13UTC from an earthquake
• ADC error on IOPSEIH45, had to do a model restart

Naoki, Camilla.

Noted by Vicky in alog 69464, the SRCL offset change significantly effected SQZ behavior. Rather than blindly having the THERMALIZATION guardian change the SQZ angle from 195 to 175 degrees 69470, we are going to repeat the SQZ angle optimization in alog 69055 during this lock.

I've commented out the lines that change the sqz angle and reloaded THERMALIZATION guardian. 

Due to the recent issues we've seen with ITMY mode8, I've unmonitored the filter engagement on its filter bank and the gain channel so that making adjustments to its phase or gain won't kick us out of observing.

EJ, Erik, TJ, Dave, Fil

At 12:38 PDT all models on h1seih45 stopped running.

There was an ADC issue, so both the front-end computer and its IO Chassis were power cycled.

When the system came back up, the 3rd ADC was showing it had failed its auto-cal, we decided to replace this ADC.

J. Kissel

Now that we've settled on a SRCL offset of -290 ct (-2.8 nm = -1 deg) LHO:69402 and DARM offset of 9.7 ct (~6 pm = 40 mA on the DCPDs) LHO:69358, I've used Evan's code from LHO:69301 (which now lives in the Calibration group's git repo common scripts project under process_sensing_darm_comb.py) to process three thermalization periods over the past few day's worth of lock acquisitions, whose times and duration are listed below.

Period  UTC Start             UTC Stop    GPS Start      GPS Stop     Segment             Feature Plot                   Plot collection
        YYYY-MM-DD                                                    Duration (hours)
(A)     2023-05-09 01:15:00   04:30:00    1367630118     1367641818   3.25 (3h 15m)       2023-05-09_0115-0430UTC        plots-1367630118-1367641818.pdf

(B)     2023-05-10 08:41:00   10:12:00    1367743278     1367748738   1.51 (1h 30m)       2023-05-10_0841-1012UTC        plots-1367743278-1367748738.pdf

(C)     2023-05-10 11:19:00   14:00:00    1367762418     1367752758   2.68 (2h 40m)       2023-05-10_1119-1400UTC        plots-1367752758-1367762418.pdf

To get a big picture feel for what's happening, first open up the 2023-05-10_0700-1600UTC_IFOStatus_trend.png which shows the following metrics of IFO status during periods (B) and (C):
    - the ISC_LOCK guardian state. Recall that 600 is nominal low noise.
    - the power in the arm cavities, show our best (most sensitive, and consistently available) metric for thermalization, where "thermalized" is ~430 kW.
    - The 410.3 Hz PCALY calibration-line informed relative optical gain, \kappa_C, where we see a value of 1.0 at the beginning of the thermalization period, and 1.024 (2.4% high) once thermalized
    - The410.3 Hz PCALY calibration-line informed coupled cavity pole frequency, f_cc, where we see a value of ~445 Hz at the beginning, and 435 Hz once thermalized.

Then, open the three period's worth of "Feature Plots." This shows the sensing function *residual* as a function of time -- the direct measurement of the sensing function (informed by new-ish, temporary-ish CAL_AWG_LINES calibration lines; LHO:69284) divided by the reference model of the sensing function (informed by the latests pydarm_H1.ini parameter set LHO:69332). The reference model is also compensated for the reported change in \kappa_C and cavity pole frequency "time-dependent correction factors" or TDCFs mentioned above (though their values are taken from the GDS pipeline's production from the 410.3 Hz calibration line rather than the CAL-CS answer in the trend). 

Each trace is a 2 minute time period.

All three thermalization periods report the same thing: since the fixed SRCL offset is intending to compensate for a large *anti*spring when the IFO is thermalized and by making the sensing function flat, it's perhaps no surprise that before thermalization -- when the cavity pole is high at ~450 Hz (and I conjecture that when the low-frequency sensing function is still flat, with no detuning) -- we see a large pro-spring at the beginning of the thermalization period until the IFO thermalizes.

The other thing that's sad and consistent is that the data showing what would be unphysical phase evolution *if* the only thing that was going on was a "simple" spring detuning evolution from pro-spring to tuned. This is, equally sadly, consistent with the pro-spring detuning that we've seen in O3 -- see the breakdown of measurement compared against the mathematical equations for pro- and anti- springs in LHO:48083. No surprise, but -- this continues to support that there is more physics than just a detuned optical spring happening here in the low-frequency sensing function.

At least the thermalized answer appears to be landing on the same location, which is consistent with what we've been seeing with the long sweeps that we run after thermalization.

Unfortunately, the data also shows that the thermalization is not consistent between these three periods -- there's shows a much larger magnitude pro-spring detuning than (B) and (C), and the starting phase is different by ~5-10 deg.
    

There are much more interesting plots in the .pdf collections for your perusal, showing
    (1) The live measured sensing function compared against the reference model (before correcting the reference model by the \kappa_C and f_cc TDCFs)
    (2) The sensing function residual, but unlike the feature plot, the reference model is *not* corrected by the TDCFs
    (3) A repeat of the feature plot
    (4) The live measured *response* function, PCAL / DARM_ERR = (1+G)/C, using *all* the PCALX and PCALY lines, compared against the modeled *response* function
    (5) The ratio the measurement vs. model in page (4), without correcting for TDCFs. This, we affectionately refer to as the response function "systematic error," or the "systematic error in the calibration"
    (6) Same as (5) but *with* correcting for TDCFs
    (7) A display of the live computation the "systematic error" that's been recently commissioned, from LHO:69285
    (8) A time-series report of the TDCFs that were used during the period.

The study leaves a lot to be desired for the next steps, that I'll continue to work on.
    (1) In addition to the 3-dimensional Bode feature plot, that shows the sensing function residual evolution in time as an evolving rainbow of colors, we want a time-series trend of the answer for the full time stretch as well. This will give a better feel the exponential nature of the evolution.
    (2) We'd really love to see more lower frequency data points, to really resolve the spring
    (3) We'd like to validate the processing of the sensing function with these continuous lines in the same way that we validated the broadband injections from LHO:69402

With lots of help from JamieR, GabrieleV, MaddieW, and AaronV, I think I've maybe gotten a better handle on applying appropriate phase advance / delays to the NonSENS training, for online cleaning that creates the _CLEAN channel. 

A caveat is that we haven't had a nicely thermalized IFO this afternoon to try the cleaning on, but I'm hopeful that since I trained everything on a thermalized IFO from last night, that once the IFO thermalizes the subtraction will be good.

For tonight, I am letting the subtraction subtract, so I've set the NOISE_CLEAN guardian to send signal out to the GDS calibration pipeline.  I think I've accepted everything in safe and observe sdfs, so that Tony will have no problem going to Observing later.

A caveat is that some of the training needs more time, including this LSC training result attached, showing that I'm expecting I'll be injecting noise below 25 Hz, but cleaning things up above 25 Hz. This is noise injection something that requires improvement over the next few days, and is a work in progress.