The BRSY measured spot is getting close to the edge of its ccd. The driftmon is about 12000 cts, out of +/-16k. This doesn't necessarily indicate that we have 4k cts range left, it's probably something more like 2k. I think that because the reflected spot is headed away from the reference spot, it may not be so bad if it drifts off the edge.

This became a problem because on Tuesday I found a gap under one of the sides of the BRSY box when I was plugging in the thermocouples. Now that I've closed that gap, the box has heated up .4 deg C. That is about 3k counts and in a bad direction. Don't know how long the gap was there, but probably for a while. Attached plot shows about Monday to now, the big increase in the driftmon and brs temperature are after I closed the gap. The VEA temp average has been more or less stable over this time. I'm also not applying any voltage on this BRS, and that wouldn't help at this point anyway.

TITLE: 12/05 Day Shift 16:00 – 00:00 (00:08-16:00), all times posted in UTC

STATE of H1: Observing

INCOMING OPERATOR: Patrick

SHIFT SUMMARY: Weak superevent candidate, followed closely by sudden lockloss. Light commissioning took place before and after locking. Locked for an hour, just started Observing.

LOG:

18:03 (10:03) Karen to MY

18:10 (10:10) Vlad to Optics Lab

18:56 (10:56) Going to Commissioning for some BRS testing by Jim

18:59 (10:59) Back to Observing

19:22 (11:22) Karen leaving MY

22:02 (14:02) Sudden lockloss. Doing some light ToO commissioning work

22:10 (14:10) Travis to Optics Lab

22:11 (14:11) Done with commissioning, beginning lock acquisition

22:13 (14:13) Travis out of Optics Lab

22:42 (14:42) Richard to CER

22:45 (14:45) Richard out of CER

13:12 (15:12) At NLN, starting commissioning

23:38 (15:38) Vlad, Dripta, Rick to Optics Lab

23:53 (15:53) Commissioning complete, going into Observe

J. Driggers, J. Kissel, N. Lecoeuche, T. Shaffer

After the most recent lock loss, we noticed the the Beam Splitter suspension was constantly saturating. We found that it was the optical lever damping loops saturating the DAC, and then going unstable. It was not until we got some part of the way through the lock acquisition sequence again that we understood why: the beam splitter's M2 coil driver state is not set back to high range (state 2) from low noise (state 3) until the ISC_LOCK manager requests the ISC_DRMI subordinate to "PREP_DRMI." With the coil driver analog filter in low noise, the digital compensation filters are boosting the digital request such that the overall transfer function to optic is flat, and thus saturating the DAC, when otherwise, in state 2, the optical lever damping control signal is well within the DAC range.

The attached time series shows 
    - the optical lever damping control request, prior to the compensation filters,
    - the request after the compensation filters
    - the coil driver state
    - the ISC_LOCK and ISC_DRMI state number.
from the initial lock loss from NOMINAL_LOW_NOISE, and then the trickle through the sequence as TJ was commissioning the ALS automation, and then once ISC_LOCK hits 101 (which is ACQUIRE_DRMI_1F) and asks ISC_DRMI to go to 10 (PREP_DRMI), the coil driver state changes from 3 back down to 2, and the DAC output returns to acceptable levels.

Also, in our finding of this switch back to state 2 in ISC_DRMI PREP_DRMI, we see that in August 2019, the *collection* of coil driver switches back to high range with PRM, SRM, and BS were moved from ISC_DRMI's DOWN state (where, honestly, we expected it to be today), to PREP DRMI because of some occasional confusion when initial alignment is run (see LHO aLOG 51325). 

We don't want to move the request to go to high range for PRM and SRM out of the PREP DRMI guardian, because later in the ISC_DRMI, they are switched to an even more different state (state 1), and if we're in a situation where we're only using the DRMI guardian *and* we loose the DRMI lock, then we want these SUS to have their states reverted back to acquire.

However, the beam splitter state is 
    - NOT changed anywhere else in ISC_DRMI, 
    - only changed to the low noise state in the ISC_LOCK guardian state LOWNOISE_COIL_DRIVERS, *and* 
    - it's got an optical lever loop constantly driving the M2 stage, where the SRM and PRM don't
so we've pulled *specifically* and *only* the beam splitter's switch back to state two out of ISC_DRMI, and put it in to the DOWN state of ISC_LOCK, right around where the QUAD's states are reverted from the changes made in the LOWNOISE_COIL_DRIVERS state, such that things are happily self-consistent. 
The PRM and SRM switches remain in ISC_DRMI's PREP_DRMI.

Upon saving the changes, we:
    - ran guardutil print ISC_LOCK for a crude check to see if we've made any python syntax errors
    - loaded these guardian changes in to both guardians
    - committed the changes to the SVN

This was also a useful command when looking through guardian logs for a given time:
    zot$ guardctrl log -a 1259618520 ISC_DRMI -n 500 | grep DOWN | grep REQUEST
    2019-12-05_22:02:17.950865Z ISC_DRMI REQUEST: DOWN
    2019-12-05_22:09:40.903001Z ISC_DRMI REQUEST: DOWN
    2019-12-05_22:23:27.785163Z ISC_DRMI REQUEST: DOWN 
    zot$
where the "-a GGGGPPPPSSS" flag is the GPS after which you want to start looking, the "-n NNN" flag specifies how many lines you want, and then you specify the guardian node. In this particlar command, I then grepped that out put for when a request was made for DOWN, but, of course, you don't have to do that.

Light commissioning after lockloss (SEI tests, green arm lock automation)

Attempt #1

Locking arms green ran automatically

Find IR ran automatically

Lockloss from ACQUIRE_DRMI_1F

Attempt #2

Everything ran automatically

NLN

The CW group noted in a recent alog (53439) and an email thread that one of our ADS lines in particular is extremely close to a pulsar that could be detected, and they reminded me that Keith Riles long ago put in an alog about what frequencies to avoid putting lines in at (alog 14836*).

The first attached plot shows a long average of the H1 DARM curve during a recent NomLowNoise stretch, with vertical lines overlayed to indicate the current values of our ADS lines, and with shaded regions to indicate where Keith's analysis says it should be okay to put lines (I use the 0.1 Hz veto regions for 2f, which I believe means that the non-green regions in my plot are within +\- 0.1 Hz of the 2f frequency of a pulsar). 

The second plot is similar, but it is a shorter average since I wanted NomLowNoise time when the ADS lines were off.  Here the green regions are the same as the previous plot, and the vertical lines indicate my candidate values to move the ADS lines to.  The red 'shoulders' on each of the ADS lines are +\-0.1 Hz of the ADS candidate frequency.  I need the lines to be at least 0.1 Hz apart in order to separate them with bandpasses that aren't too narrow.  These red lines are also a semi-reasonable proxy for the width of the upconversion shoulders on the current ADS lines, so I have put all of my candidate frequencies at least 0.1 Hz away from the edge of a green region.

This afternoon (if commissioning time is approved) I will work on moving some of the ADS lines.  My first priority will be the 22.347Hz line that the CW group called out as most 'dangerous', but if I have time I will also work on moving some of the others.  For each line that is moved I'll need to rephase and check the control loop, so I don't know that I'll get to all of them today.

* When re-searching for this alog, I find that EvanG put in an updated version in alog 44892, but my plots are using Keith's original version - I suspect that the okay regions are still fairly similar to their original values.

=== Summary ===

There were numerous tests to investigate whether the NCAL electronics causes glitches and other short duration, transient noise in DARM. The full details are given in LHO alogs 53274, 53396 however, to summarise:

1) On the 12th November 2019, a series of glitches at 30,40,50 and 60 Hz had been observed, most prominently seen on the 13th November 2019 in the Omircon summary plot.
2) The glitches persist until 14th November 2019 21:00 UTC.
3) This time coincides with when the NCAL was fully powered at EX, but not spinning.

The channels that I used to investigate the noise coupling(s) are:

Further information about the permanent PEM sensors can be found under the LHO tab at pem.ligo.org

To understand the wiring set-up of the NCAL, the wiring diagram cartoon in DCC D1900015 or a pin to pin drawing in DCC D1900499.

From the tests I have conducted, using the NCAL electronics, I am unable to reproduce the glitches as seen in the OMICRON or HVETO summary page plots, and there are otherwise no short duration, transient noises. From this I conclude that the NCAL electronics is not the primary source of the glitches and that it is acceptable to leave the NCAL on/powered during observing however, the NCAL will NOT be spinning during observe segments. The NCAL will be spun during either calibration/commissioning times for measurements or during down periods for engineering work (such as Tuesday maintenance).

It would be helpful if the DetChar team could aid in the conclusion I have drawn by using better data analysis techniques to confirm or deny this result.

=== Understanding the Original Glitchy Data ===

It is hard to pinpoint where the glitches are coming from however, the Hveto tab on the summary pages  points to H1:PEM-EX_ADC_0_18_OUT_DQ as the winning channel associated with these glitches. This channel is connected to a voltage monitor at EX for the ESD drive system indicating that the noise is coming fromt the X End station. Currently, I have not used it in my analysis-more as an indication that the noise is coming from EX however, I will add this to my analysis later. I took a four hour period of time, during 13-Nov-2019, starting at 11:00 UTC (just prior to the time period identified as bad) and tried to use LigoDV-web spectrogram and Q-Transform features in order to understand the morphology of the glitches. By inspection of the Omicron plot, although the glitches appear to be constrained to a given frequency band, they do not appear to be periodic. To me, this eliminates clock-like processes such as blinking lights/LEDs etc. As an example, I show an Omicron plot from the 13 November 2019 at 12:00 UTC and the corresponding spectrogram of DARM using the same channel. A Omega scan of the feature seen in the spectrogram between minutes 18-25 shows a loud broadband glitch at around 13-11-2019 at 12:19:07 UTC. A collection of all the plots I made and the settings I used can be found in G1902232 in addition to the results of the investigations given in the sections below.

Below we try to replicate any of these previously seen glitches in spectrogram(s) of further follow-up tests using the NCAL electronics (because we've not figured out how to create omega glitch-grams).

=== On/Off Tests ===

In order to determine if the optical encoder system was causing the issues, we turned the the encoder system off and on in ~20 minute cycles (LHO alogs 53441, 53524). In both cases, I was at the X end whilst the LHO IFO was in Nominal Low Noise, and I would carefully walk over to the BRS Heater/NCAL Motor Controller rack and flip the +/-15V switch. Only the encoder system is connected to this switch therefore, power to other system will not be affected (there are no other systems that require +/-15V however, the power is supplied by the BRS heater +/-18V system). Furthermore, in both cases, the wi-fi at EX was on and the lights were off. It is not easy to turn the power off to the NCAL Beckhoff system because, it is sharing power with the BRS Heater system therefore, turning off the power supply to the NCAL Beckhoff system would also turn off the power to the BRS Heater thus, this test was not conducted. The goal of the on/off tests is to have definitive on and off states of the encoder system within the same lock stretch and statistically, from the original glitches observed during 12th-14th November, a glitch should occur. Forcing the system on and off may force an electronics glitch and it is expected clearly see the on/off points in DARM.

During the first on/off test (LHO alog 53441), the sensor correction was disabled therefore, frequent scattering glitches swamped the spectrum making it hard to draw conclusions from the data. This is because, the magnitude of the scattering shelf glitches are much higher than the noise expected from NCAL electronics. There may be some useful information in the data however, it may be hard to find a quiet time between glitches to obtain reasonable SNR in DARM to see the NCAL electronics glitches.

During the seconds on/off test (LHO alog 53524), the sensor correction remained on, reducing the the scattering shelf glitches in magnitude and occurrences. When taking a spectrogram of this time, there is no indication, that I can see, of the glitches as a results of the On/Off tests either looking at a 0-1000Hz range or zoomed in to 0-70Hz range. in the portable magnetometer channel, the winning channel that illuded to the glitches or in DARM. During the second round of On/Off tests, the portable magntometer was on top of the encoder satellite box therefore, this sensor should show whether the NCAL encoder system glitches with the power.

=== Extended Encoder Off Time ===

To determine if the Beckhoff electronics were the cause of the glitches, we turned off the encoder system for an extended period of time. By eliminating the encoder system, the only electronics contribution would be from the Beckhoff system. The long off time would allow for sufficient integration time in order to investigate if there are any new lines as a result of the NCAL Beckhoff system or correlation with glitches. The difficulty here is that there are no fast voltage/current monitors therefore fast glitches may be missed by these monitors. 

During the times when the encoder was off and, the NCAL Beckhoff system is on, the rate of glitches appear to be the same before and after the NCAL systems' installation. From looking at the glitches summary page on the summary page, the same channel (H1:PEM-EX_ADC_0_18_OUT_DQ) remains in the top 10 winners.

=== Extended Encoder On Time ===

With the above investigations completed, we were confident enough to leaving the NCAL electronics (including the encoder system) on during observing segments. The NCAL would not be spinning however the MEDM interface would enable remote operation of the NCAL. The motivation for this was to gather enough days worth of data in order to analyse DARM over long integration times. This was to assess the impact the NCAL electronics would have on long duration searches such as CW. The results of this are currently unknown however, assistance with this would be helpful since, I am unfamiliar with long duration search techniques. Evan Goetz has started an inital investigation (LHO alog 53666) and suggests that the NCAL electronics, not sure which system, is adding more lines into the 10-100Hz band when intergrating over a long period time.

During the times when the encoder was on, the NCAL Beckhoff system is on and, the NCAL is stationary, the rate of glitches appear to be the same before and after the NCAL systems' installation. From looking at the glitches summary page on the summary page, the same channel (H1:PEM-EX_ADC_0_18_OUT_DQ) remains in the top 10 winners (for example 20th Nov 2019).

=== Spin Tests ===

During the Tuesday maintenance on the 26th of November 2019, R.Schofield and R.Schofield and I used the portable magnetometer to investigate magnetic coupling of the NCAL system in various locations around the XVEA (LHO alog 53503). We left the portable magnetometer on top of the satellite box for future so that we have a sensor as close to the NCAL as possible and can trend the channel H1:PEM-EX_ADC_0_11_OUT_DQ for future investigations. The goal of these tests was to determine the how noisy the NCAL system, using the magnetometers and accelerometers as our metric.

When looking at spectrograms of the magnetometers{X,Y,Z} and the accelerometers{X,Y,Z}, there are harmonics that increase in frequency as the NCAL is accelerating or decelerating. When the NCAL is at rest or is at a constant velocity, there does not appear to be loud glitches in the magnetometers however, the accelerometers show glitches throughout the test period. Since I was in th XVEA during the times of these tests as well as R.Schofield undertaking other tasks in the XVEA, these glitches may be associated with out movements rather than the NCAL system.

=== NCAL Electronics Timeline ===

For follow up data analysis, the current electronics timescale of NCAL is as follows:

=== Other Thoughts ===

If the power supply, that provides power to the NCAL electronics, glitches then this may couple though the NCAL electronics into DARM. To my knowledge, there is not a monitor on the power supplies and there are no fast current/voltage monitors in the NCAL electronics path. If there is a fast current/voltage monitor on the BRS heater electronics then this may be an indicator of the power supply state however it would be hard to draw conclusions without another reference.

Ops Shift Transition: 12/05/2019, Day Shift 16:00–00:00 (08:00-16:00) - UTC (PT)

State of H1: Locked

Intent Bit: Observing

Weather: 0-5 mph wind

Primary 0.03 – 0.1Hz: 0.01 um/s

Secondary 0.1 – 0.3Hz: 0.4 um/s

Outgoing Operator: Jeff

Quick Summary: Locked and Observing for 1 hour

   While working on relocking, noticed the "Fast Shutter Failed Test. HV Not Ready" message. This was after 13:00 (05:00) and Richard was due in in less than an hour, so opted to wait for his arrival. When Richard arrived, we went into the LVEA to look at the Fast Shutter Driver and Shutter Controller. Reset these and then took a look at the HV Interlock for HAM6. (See aLOG #53697). Richard got the Interlock to reset and I started to relock. 

  14:09 (06:09) Started relocking
  14:13 (06:13) ACQUIRE_DRMI_1F
  14:18 (06:18) ACQUIRE PRMI
  14:20 (06:20) PRMI _LOCKED
  14:24 (06:24) DRMI_1F locked
  14:51 (06:51) Back at NLN
  14:52 (06:52) Accepted 2 ETMX SDF Diffs and went back into Observing 
         

   Lost lock at 11:37 (03:37). Working on relocking. 

V. Bossilkov, S. Karki, J. Kissel, T. Mistry

We've gathered an entire set of calibration measurements today (Full Actuation and Sensing Function Suite), as well as gathering several bonus measurements, exploring 
    - the high frequency dynamics of the UIM (now that we've identified that they are indeed important) 
    - a "sweep" of the NCAL system, to see how the NCAL excitation looks as a function of frequency
    - whether the L2A2L coupling is non-linear 

This'll be days work of analysis work, so stay tuned for future results.

For now, here're the templates for the standard measurements, I'll leave it to Timesh, Vlad, and Sudarshan to log there files.
Sensing Function: 
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs
    2019-12-04_H1_DARM_OLGTF_LF_SS_5to1100Hz_15min.xml

    2019-12-04_H1_PCALX2DARMTF_LF_SS_5t1100Hz_10min.xml
    2019-12-04_H1_PCALY2DARMTF_LF_SS_5t1100Hz_10min.xml

    2019-12-04_H1_PCALX2DARMTF_BB_3min.xml
    2019-12-04_H1_PCALY2DARMTF_BB_3min.xml

Actuation Function: 
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOActuationTFs/
   2019-12-04_H1SUSETMX_L1_iEXC2DARM_8min.xml
   2019-12-04_H1SUSETMX_L1_PCAL2DARM_5min.xml

   2019-12-04_H1SUSETMX_L2_iEXC2DARM_12min.xml
   2019-12-04_H1SUSETMX_L2_PCAL2DARM_6min.xml

   2019-12-04_H1SUSETMX_L3_iEXC2DARM_12min.xml
   2019-12-04_H1SUSETMX_L3_PCAL2DARM_6min.xml

#ThesisWork!

Since the NCal electronics have remained on since GPS=1258930068 (Wed Nov 27 22:47:30 UTC 2019), see LHO aLOG 53524, I have taken the weekly Fscan averages from the summary pages to look for any additional noise from spectral artifacts. Granted, this is not a conclusive study to confirm or vindicate any couplings caused by NCal, but it is a first look at long duration averages and suggests we should investigate this further. I would also suggest that it's worth turning off the Ncal as long as it does not remain an essential tool for IFO operations. Past experiences indicate that all non-essential electronics should be disconnected during observing runs. This suggestion is, at the moment, due to an abundance of caution regarding new, non-essential IFO operations electronics connected during an observing run.

Below I attach three ratios of the weekly Fscan averages of H1:GDS-CALIB_STRAIN in the 10-100 Hz band, a common region for noise lines from electronics.
Figure 1: ratio of Fscan weekly average Nov. 3-10 / Nov. 17-25
Figure 2: ratio of Fscan weekly average Nov. 10-17 / Nov. 17-25
Figure 3: ratio of Fscan weekly average Nov. 25-30 / Nov. 17-25 <-- NCal electronics remain on from Nov. 27

In the two weeks preceding the Ncal being left on, the Fscan ratios show some variations, as indicated by the ratio deviating from a nominal (unchanged) value of 1.0. These variations are below a maximum ratio value of 20 (one line only). See figures 1 and 2. However, the final figure shows the change from the week the NCal electronics remain on and many lines show large ratio values up to maximum values greater than 50.

Further study is needed to determine if this really is caused by the NCal electronics remaining on since Nov. 27. At the moment, I'd suggest turning the electronics off and possibly disconnecting as well to be fully cautious.

J. Kissel

As per our studies on Monday, Keita advised that we engage at least some level of analog low-pass on the QUADs' UIM coil drivers (see LHO aLOG 53376, and subsequent comments.)

After we lost lock from an obscene amount of wind, I've now switched the ETMs to be in state 2, with *one* low pass on. This is the state I studied in LHO aLOG 53486.

For ETMY, it was as easy as requesting state 2 and accepting it in the SDF system. However, for ETMX, I found that the ALS_DIFF guardian was requesting state 1 after every lock loss. So, I've changed the hard-coded request (which lived in the DOWN state), to be to request state 2. And also accepted this in the SDF.

It looks like someone has already requested that the ITMs are in state 4 (with all three z:p = 10:1 filters ON), and accepted as such in SDF. These have been ON throughout the last ~24 hour lock stretch that started at the end of maintenance yesterday. State 4 should be fine as far as actuator saturations because we never send any drive request to the UIM stage of the ITMs. As such, turning on ALL low passes will only filter the DAC noise which is what we want.