TITLE: 06/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 6mph Gusts, 4mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.10 μm/s
QUICK SUMMARY: Starting Maintenance day. Ryan has taken the SEI_CONF to Maintenance, IFO has stayed locked so far.

Dust monitors, VAC, SUS, SEI, CDS okay

TITLE: 06/13 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:

• Lockloss at 13:16UTC
• We weren't able to get either the PEM magnetic injections or the inlock charge measurements due to the lockloss
• I keep getting booted out of the alog when I try to save to draft?

Lock #1

• Locked for 6:31
• DCPD and EX saturation at 12:55UTC
• Lockloss at 13:16UTC Potentially from an EQ

Lock #2

Both arms went through increase flashes, Xarm took a while (10 mins)

Couldn't catch DRMI, or PRMI and it was clear from AS AIR that something was badly aligned so I ran an initial alignment. DRMI still struggled a bit despite good flashes and took about 5 minutes to lock.

NLN at 14:51

LOG:                                                                                                                                                        

Lockloss at 13:16UTC, we were getting hit by a 5.4 from Papau New Guinea, seismic was starting to increase. Verbal did not say if SEI_CONF went to EQ mode

Control room workstations were updated and rebooted, including opslogin0 (nomachine).  Only operating systems were updated.  Conda environments were not affected.

STATE of H1: Observing at 137Mpc

We've been locked for 4:18, in observing for 4:07

We had an EX saturation at 10:19:45UTC

No alarms, 1 very small EQ ~2hrs ago (max of 165 on Peakmon)

TITLE: 06/13 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 132Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 7mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.10 μm/s
QUICK SUMMARY:

Taking over from Austin

• Locked at NLN since 06:42UTC, In observing at 06:53UTC
• CDS/Wind/Dust looks good

TITLE: 06/12 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 133Mpc
SHIFT SUMMARY:

- Lock #1:

• Acquired NLN @ 21:40, observing @ 22:06
• EX saturation @ 23:16
• Lockloss @ 0:07

- Lock #2:

• ALS/IR was unaided but there was a lockloss when transitioning to DRMI from PRMI - failed twice, but caught the 3rd time
• Acquired NLN @ 1:29, OBSERVE @ 1:45
• PI 31 ringup @ 1:43 - guardian looks to have damped it
• Taken into Commissioning @ 1:55 due to TCS SDF changes - ss attached, eventually these changes reset on their own, back to Observing @ 1:57 - Tagging TCS
• Lockloss @ 4:35

- Had issues with ALS locking, so redoing an IA, during which it seems the X beatnote was poor again, so I reset the crystal frquency to get the beatnote to ~39 MHz, that seemed to have fixed the issue, rest of IA went fine

- Lock #3:

• Still having issues with the X beatnote, preventing me from locking ALS
• ALS X is going into fault reading PLL - looks like the the PLL lock would be green, the flash to red, then back to green very quickly on the PLL overview, but nothing is sticking out as being the cause of this
• After looking through some alogs, I used this log, and set the beatnote rf minimum to -15 to see if that would help things
• Looks like I can now get up to CHECK_IR, ALS_COMM found ok and ALS_DIFF was fine tuning before we lost lock again (both the COMM (-6) and DIFF (-6) beatnote were good)
• The second time around, again I set the beatnote rf minimum to -15 and was able to continue locking - Tagging ISC
• Acquired NLN @ 6:42, OBSERVE @ 6:53

- Leaving H1 to Ryan C. in observing

LOG:                                                                                                                                                                                                                                                                                                                                                         

LOCKLOSS @ 4:35, not seeing a whole lot of ASC movement, but am seeing some motion in LSC loops - scope attached

Following a lockloss, H1 has now been locked and in observing for ~1:30, all appears nominal.

The Following features have been removed from the Vacuum screen on nuc22 found at /opt/rtcds/userapps/release/cds/h1/scripts/fom_startup/nuc22/H0_VAC_SITE_OVERVIEW_CUSTOM.ui and H0_VAC_SITE_OVERVIEW_CUSTOM.adl upon the request of Gerardo.

Neg pumps pressure transducers at the corner station: PT-191, PT-192, and PT-193.
Neg pumps pressure transducers at the Y-End station: PT-426 and PT-428.
Neg pump pressure transducer at the X-End station: PT-526.
The corner station "CP1 IP" and "CP2 IP".

The old versions still exist and have been renamed with an .adl.old file extension. For the MEDM screen that Sitemap uses the file lives here /opt/rtcds/userapps/release/vacuum/h0/medm/Target/H0_VAC_SITE_OVERVIEW_CUSTOM.adl

LOCKLOSS @ 0:07, had a SRM saturation right before the lockloss. Seeing some movement in INP1 P - scope attached

We have had several locklosses recently that can probably be attributed to a CSOFT P ring up. Starting on 6/8, I looked through all the locklosses and found 6 in total that had a CSOFT P ringup with the oscillation at 0.45 Hz. Some of these have already been noted in various alogs (shout out to Camilla for pointing them out here 70267).

CSOFT P locklosses:
6/8 2:45 UTC

6/8 6:53 UTC

6/10 6:53 UTC

6/10 19:50 UTC

6/11 21:45 UTC

6/12 12:12 UTC

The operating power for each of these locks seems to be about 430 kW in the X arm and 432 kW in the Y arm +/- 1 or 2 kW which is pretty standard, so it does not appear to be power related (at least lock to lock). The TMS servo signals also look stable and we are staying well centered on the TMS B QPDs and are not very far off center in the A QPDs.

Each of these locks have a different durations ranging from 2 to 9 hours, and there have been several other locks in this period where the locklosses are not attributed to ASC. Whatever the problem is, it does not appear to be consistent from lock to lock

We haven't had ASC problems for a while. These are some main differences from our known-to-be-stable configuration that could affect the ASC: ring heater change on EX, 1 W drop in input power, new input alignment from PR3 move.

One thing we could try is increasing the loop gain slightly and seeing if these locklosses persist. Right now the EPICs gain is set to 20. I have been able to increase this gain by up to 3 dB in the past without issues.

TITLE: 06/12 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 133Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 11mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY:

- IFO is locked and in observing as of 12:41

- CDS/SEI/DMs ok

TITLE: 06/12 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 133Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 12mph Gusts, 10mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:

Lockloss at 16:13 UTC

Started trying to immediatley relock.

Lockloss at ACQUIRE_DRMI_1F 16:29 UTC.

Starting Initial_Alignment 16:30 UTC.
Initial Alignment went smoothly.

Relocking went smoothly as well.
Arrived at NOMINAL_LOW_NOISE at 17:33 UTC.
 
Had a Strange SDF Diff H1:ISC-RF_Y_AMP71M_OUTPUTMON Dr.  it was 20.5 and is now 19.5.  Dr.Sigg Accepted it.
OBSERVING Reached at 17:46 UTC

Unlocked Due someone Clicking an MEDM button they though was a safe button to click, but it was not.
See alog https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=70375

Reached NOMINAL_LOW_NOISE at 21:40 UTC
Shelia, & Brina did an excitation test for the gain frequency before the IFO was taken to Observing mode.

Observing 22:06 UTC

Current IFO Status: NOMINAL_LOW_NOISE & OBSERVING                                                                                                                     

Earlier Lockloss from unknown cause.
Lockloss from NLN:
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1370621620

I tried trending a number of channels back that seemed like they had a large magnitude or interesting abbiration, But when compairing them to live IFO channels they seem to be channels that usually have larger magnitudes.

Even after the Analysis finished there isn't an well defined reason for the lockloss that is obvious to me.
 

Since we are keeping track of the changes to PRCL, the lockloss today (6/12) at 2:02 UTC might be due to PRCL. There is an oscillation in all three LSC loops around 13 Hz right before lockloss.

Following up on EJ's alog, I have extended the code to check if the unresponsive filter is under local control by the model. I ran it this afternoon while H1 was in observe, running in 'nice' mode so as to not hammer the frontends with CA requests (delays between requests).

If we disregard locally controlled filters, the number of unresponsive filters is reduced to 3 (see attached). The top one runs on h1lsc, the other two on h1lscaux.

Here is the full list, with locally controlled filters marked with *

Total numer of filtermodules = 13367
num unresponsive 14

h1lsc {'LSC-EXTRA_AI_2': 'FM2  '}

h1lscaux {'LSC-LOCKIN_1_DEMOD_9_I': 'FM1  ',
          'LSC-LOCKIN_1_DEMOD_9_Q': 'FM1  '}

h1sqz {'SQZ-RLF_VCXO_SERVO': 'FM2* '}

h1alsex {'LSC-X_ARM_DRIVE': 'FM3* FM5* '}

h1alsey {'LSC-Y_ARM_DRIVE': 'FM3* FM5* '}

h1sussqzin {'SUS-ZM1_M2_COILOUTF_LL': 'FM1* FM6* ',
            'SUS-ZM1_M2_COILOUTF_LR': 'FM1* FM6* ',
            'SUS-ZM1_M2_COILOUTF_UL': 'FM1* FM6* ',
            'SUS-ZM1_M2_COILOUTF_UR': 'FM1* FM6* ',
            'SUS-ZM3_M2_COILOUTF_LL': 'FM1* FM6* ',
            'SUS-ZM3_M2_COILOUTF_LR': 'FM1* FM6* ',
            'SUS-ZM3_M2_COILOUTF_UL': 'FM1* FM6* ',
            'SUS-ZM3_M2_COILOUTF_UR': 'FM1* FM6* '}
 

This has been kind of hard to pick out, but it definitely seems like the IFO is less robust now against earthquakes that it was before increasing power over 60W in the IMC. Basically since then, we haven't been able to stay in NLN for any earthquakes over ~.5um/s, as measured by the peakmon eq witness channel.

Attached plot shows peakmon vs lock state trends since Feb 10 this year. The blue trace is the peakmon ground velocity minute trend, each green point indicates where the IFO was still locked 2 minutes after a local maximum ground velocity (found using the matlab peakfinder routine with some prominence and time separation requirements). The two marked points indicate where IMC power was increased above 60W (X=79730 minutes after Feb 10th) and the start of the run (X=148255 minutes after Feb10). Before power up we had multiple locks survive multipe 500nm/s eqs, a few around 1micron/s and one at almost 3micron/s. After the power up, the IFO doesn't ride out any eqs over 500nm/s and we basically not survived any notable eqs since the start of the run.

This is further reinforced by TJs alog from earlier today, when SEI_CS transitioning knocked us out of OBSERVE. Unless SEI_CS got dropped off the exclude recently, I suspect we never noticed because we were losing lock before the seismic system switched.

There have been no changes to the SEI eq controls. I looked at the small eq Camilla noted this morning and didn't see anything suspicious in the seismic systems, but I'll keep digging.

I looked again at DARM at 60W vs. 75W, comparing sqz and no-sqz for both. Each DTT trace uses 8 seconds, 50% overlap, 100 averages.

Lighter colors. 60W times used from 68251, w/15dB generated sqz.

• cyan,  FDS    @ 60W -  1364158671
• yellow, No SQZ @ 60W -  1364159804

Darker colors. 76W times from 6/4, ~3 hours into lock (semi-thermalized, past fast initial thermalization).

• red,    FDS    @ 76W  - 6/4 18:25 UTC
• purple, No SQZ @ 76W  - 6/4 19:00 UTC

See the summary page with this time window-- comparing red vs. black in this screenshot, I don't see obvious + excess noise with the squeezer on below 50Hz, when we are past fast initial thermalization.

Re: "fast initial SQZ thermalization" -- For reference, see the V-shape in blue that DARM takes with the fast 0.5-1 hour SQZ thermalization from a cold IFO (even unlocked for just 1-2 hours). Notably, in the first 0.5-1 hour, esp if the IFO was cold, the fast-thermalizing-SQZ (blue) exceeds the no-sqz noise (red) both below 50 Hz, and above 1kHz. I think this is related to the rapidly thermalizing SRCL detuning, which rotates squeezing across the band for the first 0.5-1 hour. To fix this SRCL detuning problem for sqz misrotation, I think we'd need to thermalize the SRCL offset; it wouldn't be sufficient to thermalize the sqz angle. If we have to pick an angle, we should pick the sqz angle that optimizes the bucket noise ~100 Hz; this is the most stationary as SRCL thermalizes. I think for calibration, Jeff sees this too as wildness in the rapidly thermalizing sensing function at the start of locks. This effect seems more prominent from an cold IFO, as it seems more muted of a V-shape when IFO relocks quickly (not much down/cold time).

Note the excess noise 20-50 Hz in the 60W vs. 75W configuration. Brina is now looking into understanding the calibration status and LSC coherences for these sqz and no-sqz times.