We have lost several corner stations front end models in what may be a Dolphin glitch. Caused the lock loss at 16:12PDT.

TITLE: 04/30 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 130Mpc
SHIFT SUMMARY:

• Dropped out of Observing at 19:49:56 back into Observing at 19:50:14, by the time I opened up sdf the diffs were gone so I'm not sure exactly why we dropped out
• Dropped out of Observing for the CP1 fill just like yesterday at 20:15:00UTC, back to Observing at 20:18:11UTC
• Locked for 31:43!

LOG:

Sun Apr 30 13:18:03 2023 INFO: Fill completed in 3min 3secs

• All seems well, no issues, still locked at NLN (27:50) and in Observing mode since 20:17UTC yesterday
• Every few hours EX gets a small kick which we see in the suspension briefly saturating, the ADS signals which diverge for a few seconds before coming back down, in the OAF signals, and in the DCPDs. Glitches?
• My work stations been extra laggy today (cdsws13)

TITLE: 04/30 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 127Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.22 μm/s
QUICK SUMMARY:

• FMCS RO_WATER was alarming when I arrived
• Still locked from yesterday (23:33), and still in Observing
• Low wind, seism, no dust alarms

TITLE: 04/29 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 129Mpc
SHIFT SUMMARY:

Stayed in NLN for the whole shift, no issues

LOCK#1:

• Relocking at MAX_POWER when I arrived, NLN at 15:14UTC waited for the ads signals to converge to turn on the camera servos (~15 minutes) before I could go to Observing
• In Observing at 15:31UTC
• CP1 overfill (H0:VAC-LY_CP1_100_LLCV_MAN_POS_PCT) briefly dropped us out of observing at 20:15UTC , back to Observing at 20:17UTC

LOG:                                                                                                                                                                                                                        

Sat Apr 29 13:17:44 2023 INFO: Fill completed in 2min 44secs

We're still locked at NLN since 15:14 (3:52) and we've been in observing mode since 15:31UTC, quiet day.

TITLE: 04/29 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.23 μm/s
QUICK SUMMARY:

• Lost lock at 14:13UTC, currently at MAX_POWER relocking
• Low wind, no alarms, low seismic

TITLE: 04/28 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:

Main Notes:

- LVEA is LASER HAZARD until at least Monday morning

- IFO is in Observing mode as of 06:47 Apr 29th UTC

- Lock #1:

- Held at ENGAGE ASC since ADS PIT/YAW 3 were not converging whatsoever

• Elenna tried doubling the gain on the ADS filter to no avail, she then tried turning off the loop and aligning PRM with the intention of turning back on the loop
• No luck, so tried aligining REFL using IM4 and PRC2 using PR2
• LOCKLOSS @ 0:23 UTC - possible alignment being too far off

- Lock #2:

- LOCKLOSS @ 0:25 - LOCKING ARMS GREEN, unknown reason

• My cue to start an initial alignment

- Lock #3:

- This lock went fairly well, acquired NLN @ 01:56 UTC

- Do have a "Diffracted power low" on DIAG_MAIN for the PSL 

• Took a look and the AOM Diffracted Power is ~1.35% - not sure what this value is, nominally

- There were differences in the observe SDFs preventing me from going into Observe but after talking with commissioners, they have been reconciled - Screenshots attached

- IN OBSERVING: 02:59 UTC

- EX saturation @ 04:41, no ringups on ASC but do see a glich/spike on the OAF BLRMS screen for all bandpasses and DCPD saturations went bonkers for a second - glitch?

- LOCKLOSS @ 04:44 (DCPD saturation on verbal), shortly after, probably due to whatever caused the glitch above

- Lock #4/5/6:

- Had to touch ALS DIFF just a hair to catch IR

- LOCKLOSS @ 05:19 ACQUIRE DRMI, then again at LOCKING_ALS, then again at LOCKING_ALS...

- Lock #7:

- GRD was able to do most of the heavy lifting but still noticing that the ADS PIT/YAW 3 going off the rails when we get to ENGAGE ASC - but after awhile it calmed down and started to converge

- Acquired NLN @ 06:20 UTC

- IN OBSERVING: 06:47 UTC

Other Notes:

- Upon finishing an initial alignment, ISC LOCK went into ERROR - see screenshot

• Error in the CHANGE_POL state - claimed that the value incflashes_time was not defined, when in actualality, it was instantiated on line 962 of ISC_LOCK
• No idea why it's not seeing that, so I'm temporary setting the error variable to an integer value of 120 (the value of incflashes_time) for now - Tagging ISC

- The ALS GRD went through CHANGE_POL and in doing so created an Observe sdf diff in SYSCSAUX, the channel in question was just a step size so I vote we unmonitor this channel (and channels alike) so we don't run into it in the future

LOG:                                                                                                                      

After having some issues relocking after a lockloss, I did an initial alignment and we are now back up at NLN as of 01:56 UTC. However, I cannot flip the Observing bit due to a few sdf diffs left. I have cleared most already but I have attached the two remaining channels left just in case anyone seeing this knows if we should accept or revert these changes.

J. Kissel, E. Goetz

Executive Summary: after thinking about things a bit more, and considering all the conflicting metrics, we need to change frequency of the PCALX 410.2 Hz calibration line, and update/revert *all* the CALCS DEMOD filters and parameters, in order to achieve a compromise between Line-to-Line Bias, IFO parameter change time scales, and accuracy in uncertainty estimate. We'll do this on Monday 2023-05-01.



The other day, when Evan and I increased the time-constant of the I & Q low pass filters in all of the calibration line DEMODS (LHO:69117), our metrics under consideration were:
    (1) We need to narrow up the band-pass for the PCALY 410.3 Hz line because we found that the adjacent PCALX 410.2 Hz line was polluting the TDCF estimates of the relative optical gain and cavity pole (LHO:68479)
    (2) As a general principle for any DEMOD process, one must make the impulse response of the I and Q low pass filters longer than the impulse response of the SIG band-pass filters, or else the DC component of the I and Q signals (the output of the low pass) is going to be entirely confused by the impulses of the filter rather than actual changes in the line amplitude and phase.

So, based on those criteria, we figured "let's just make all the low-pass filters long, i.e. with a 40 sec time constant, or 0.025 Hz corner frequency" so that regardless of whether the band pass on the PCAL lines is narrow (+/- 0.03 Hz) or wide (+/- 0.1 Hz), we can just consistently "average for longer and get a less noisy answer of the (I and Q) cast into (Real and Imag) cast into (magnitude and phase)." That's LHO:69117. Today as a part of the thinking that went into this aLOG, we confirm that Metric (2) is satisfied by comparing the step responses of the two filter pairings -- see 2023-04-28_DEMOD_Filter_ImpulseResponses.png. Indeed in both cases (red and blue), the impulse response of the SIG band-pass is shorter than the I & Q low-pass. 



But, then we remembered that we also want to compute the uncertainty in the answers we get from the TDCFs and systematic error monitoring. The uncertainty calculation depends on the coherence, C, of the transfer function as well as the "number of averages", Navg. In the frequency domain, the thing that you're averaging is consecutive FFTs. The relationship between coherence and uncertainty we've been using for decades is the ol' Bendat and Piersol formula,
    Unc [rad] = sqrt( (1 - C) / (2*Navg*C) )

Here, in the case of a live calculation, we treat the FFT length as equivalent to the time constant of I & Q low-pass filters. The critical thing to remember there is that increasing the number of averages does NOT improve the uncertainty, it improves the accuracy of the estimate of coherence or uncertainty. 

So, this adds two more metrics to consider: if we want to keep the time over which we're creating this live rolling average constant -- at ~2 minutes, or ~120 seconds -- the time scale that we think is about the right time scale to capture the speed at which the IFO thermalizes and/or changes alignment (a number we collectively settled on after an old, crude, study by Sudarshan in LHO:22753) then we should favor more averages of shorter FFTs. 
    (3) We want FFT stride * Navg to be ~120 seconds, and
    (4) in order to achieve a more accurate estimate of the coherence and thus uncertainty we want higher corner frequencies on our I and Q low-pass filters, and more averages

In the CALCS front-end code as it stands now, the FFT length and Number of averages for all DEMOD's uncertainty estimate are determined by one pair of parameters, Navg = H1:CAL-CS_TDEP_COH_BUFFER_SIZE and FFT length = I & Q Low Pass Time Constant = H1:CAL-CS_TDEP_COH_STRIDE. This means
    - whatever we choose for the I & Q filters' low-pass time constant, you MUST update the FFT Length = COH_STRIDE to match it, and
    - Once one update the FFT length = COH_STRIDE, you have to update the Navg = BUFFER_SIZE to keep the overall time period sampled constant (and in our case we want this at 2 minutes).

So Metric (4) -- wanting high corner frequencies, low time-constants on our I & Q low passes in order to get the best estimate, is in direct opposition to Metric (1) -- wanting narrow SIG band-passes to isolate the line frequencies from other lines, because of Metric (2) -- a narrow band pass has a long impulse response, and we'd need low corner frequencies, high time-constants  on out I & Q low passes.




So how do we decide on the compromise?
At the moment, we're leaning towards "it's right before the engineering run, so the newest thing that's causing problems must change."
That means:
    - Move the PCALX 410.2 Hz PCALXY comparison line further away from the pre-existing PCALY 410.3 Hz TDCF line. Joe, for entirely different reasons, recommends 0.5 Hz separation instead of 0.1 Hz.
    - Update the "DARM Model transfer function values at calibration line frequencies" EPICs records for the PCALX 410.2 Hz line,
    - Revert all DEMOD the band-passes to have a pass band that's +/- 0.1 Hz wide (what we had in O3)
    - Revert all DEMOD I & Q low passes to a 10 second time constant, or 0.1 Hz corner frequency
    - Change COH_STRIDE back to 10 seconds to match the low pass, and Change the BUFFER_SIZE back to 13.0 in order to preserve the rolling average of 2 minutes.

This is too much to do on a Friday night at 6pm local time, so we'll make all these changes on Monday morning. That'll also give some time for others to weigh in if they're leaning a different way on the compromise, or they've got other metrics we haven't yet considered.

We will not be able to make frequency noise injections at this time. We should make an effort to measure CARM through thernalization and check if the additional gain (see alog 69152) is sufficient.

A simple classic scattered light model for the 4 Hz bumps can explain the bumpy shape. For example, something moving at 4.05 Hz with a Q of ~80 seems reasonable.

We would see scattered light bumps only when the scattering element peak to peak motion is lower, for example 0.6 lambda peak. If the object is moving faster, like 1-2 lamba peak, then the scattered light noise would be broad band and without any distincitve feature.

If this were the case, we would see the 4 Hz bumps only when the scattering object is, by chance, moving slower than normal, and we would always have a continuos scattered light background. 

This would also imply that all attempts to find the origin of the 4 Hz bumps by exciting the scattering object would fail: we see the bumps only when the scattering element is by chance moving less than usual. 

TITLE: 04/28 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 6mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.13 μm/s 
QUICK SUMMARY:

- IFO just lost lock @ 23:02 UTC due to HAM1 FF changes by commissioners

- Working to get the IFO back up to NLN

- CDS/SEI/DMs ok

TITLE: 04/27 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
SHIFT SUMMARY:

- IFO is locked at NLN, but not in Observing Mode due to unreconciled observe SDFs, attached below, so I am leaving it in Commissioning mode - Tagging ISC

- Lock #1:

• LOCKLOSS @ 03:35 - I believe this one is ground motion related - possible aftershock from EQ?

- Lock #2:

• Held in READY to put it values of ISC EX/Y whitening gains to allow IR to catch (alog here) 
• After caputting all values I am still having issues...this time with the LSC_TR_Y PD not getting any light.
• Tried from the beginning - set the values in READY and this time it was LSC_TR_X not catching...
• Rerequested FAST_SEARCH and COMM was able to catch...DIFF failed again...third times the charm
• After multiple attempts at rerequesting and putting in the values, it finally caught...praying we don't lose lock again
• DRMI failed, but let it run through to CHECK MICH FRINGES and back to PRIMI ASC where it caught and moved up
• Acquired NLN @ 05:57

Other Notes:

- 2:14 UTC - inc EQ from South Georgia - 5.8 mag eq

• We did not go to EQ mode, but I guess we didn't need to? Peakmon highest values were only at 80...interesting, quite low for a 5.6
• Also interesting - @ 2:31 UTC we got a SEISMIC to CALM alert, but no indication that it ever transitioned out of CALM to begin with? Tagging SEI

- 3:25 UTC - EX saturation, IFO was a little unstable, seeing ringups in INP1_P_INMON, MICH_Y_INMON, CSOFT_P_OUT, CHARD_Y_OUT, and DHARD_Y_OUT

• Though I'm probably chalking this up to ground motion, all pickets fence seismometers are lighting up and seeing elevated motion in both seismon DMT plots - maybe an aftershock from the previous EQ?

LOG: