aLIGO LHO Logbook

H1 CDS

david.barker@LIGO.ORG - posted 09:43, Sunday 04 June 2023 - last comment - 14:17, Monday 05 June 2023(70117)

ISI Ham7 ADC signals for the second ADC, channels 8-19 are not reading the correct signal

Austin, Jim, Dave:

Starting at 06:56 PDT the h1seih7 ADC1 (2nd) channels 08-19 started misbehaving. All the other channels on this ADC are unaffected.

Austin confirmed that the AA chassis for this ADC looks correct on the mezzanine.

Jim confirmed that this block of channels are associated with a single ISI interface chassis which feeds the AA.

The six broken signals are:

H1:ISI-HAM7_L4CINF_[V2,H2]_MON

H1:ISI-HAM7_GS13INF_[V2,H2]_MON

H1:ISI-HAM7_CPSINF_[V3,H3]_MON

Comments related to this report

jim.warner@LIGO.ORG - 09:52, Sunday 04 June 2023 (70120)

Link

The asds of the CPS, GS13 and L4C in this chassis all look identically bad, the other sensors seem fine. All ISI controls are off, loops can't be re-engaged at this point and we probably can't get the squeezer back on. I don't think we can get to Observe without fixing this.

Images attached to this comment

david.barker@LIGO.ORG - 09:58, Sunday 04 June 2023 (70121)

Link

Attached ADC Monitor MEDMS for h1iopseih7 and h1isiham7 highlight the broken channels. The signals have two modes, the are either somewhat static with incorrect low values, or they are in a flashing mode, where they transition between the low value to a very high value (tens of thousands) with a flash frequency of about 1Hz, they then go back to the quiet mode for 10-20 seconds.

For about 30 minutes after the 06:58 failure these signals were very noisy all the time, and then they settled to the static/flash-modes we are seeing now.

Images attached to this comment

jim.warner@LIGO.ORG - 11:59, Sunday 04 June 2023 (70122)SEI

Link

Marc, Austin and I pulled a spare ISI interface chassis from the staging building test stand and replace the corner 2 unit. So far signals all look normal now and the ISI was able to re-isolate. Marc opened the chassis and it looks like a cap or something failed on the power board.

We also noticed at the mezzanine racks that all of the chassises there are quite hot, I think Marc said he saw 110 F on some part of a chassis. It's warm out there, maybe we should space those chassises out a bit on the rack.

We replaced S20000568 with S1201328. This spare interface is an ALIGO spare and the last one we have.

IFO stayed locked this whole time, but squeezer was down. Vicki has now relocked and we are back observing.

marc.pirello@LIGO.ORG - 12:03, Sunday 04 June 2023 (70123)

Link

C5 blown on power regulator.

Images attached to this comment

jim.warner@LIGO.ORG - 12:10, Sunday 04 June 2023 (70125)

Link

Asds of sensors after chassis swap.

Images attached to this comment

michael.landry@LIGO.ORG - 15:47, Sunday 04 June 2023 (70126)

Link

Nice recovery, well done all.

jeffrey.kissel@LIGO.ORG - 14:17, Monday 05 June 2023 (70147)CDS, FRS, OpsInfo, SEI

Link

Tagging SEI, OpsInfo, and FRS.

Opened and closed FRS Ticket 28196.

Observation time lost is ~5 hours, see attached screenshot.

Images attached to this comment

H1 General

Link

anthony.sanchez@LIGO.ORG - posted 08:09, Sunday 04 June 2023 - last comment - 09:47, Sunday 04 June 2023(70114)

Sunday OWL Shift End

TITLE: 06/04 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 9mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:

8:15 UTC 18hours of Lock!

13:56 Lockloss due to ETMX L3 Motion after 23 hours of lock.

Relocking:
I tried to just quickly relock with out an Initial Alignment, and lost lock at Locking ALS.

I then did a Initial Alignment and and started relocking again.
There were 2 Locklosses at TRANSITION_DRMI_TO_3F, but on the 3rd attempt H1 was able to make it PREP_ASC_FOR_FULL_IFO.

Current IFO Status: LOCKING.

Comments related to this report

austin.jennings@LIGO.ORG - 08:25, Sunday 04 June 2023 (70115)

Link

After the long lockloss occurred, the ISI HAM 7 WD was tripped and was not reset properly upon relocking. We are now relocking at MAX_POWER with the WD still tripped and all BLRM bandpasses are railed - ss attached, preventing me from untripping the WD. Will monitor as the BLRMs seem to be slowly going down. - Tagging SEI

Images attached to this comment

jim.warner@LIGO.ORG - 09:47, Sunday 04 June 2023 (70118)

Link

It looks like the corner2 ISI Interface chassis might have failed. Working with Dave and CDS to try to get someone out to try to replace it.

jim.warner@LIGO.ORG - 09:47, Sunday 04 June 2023 (70119)

Link

It looks like the corner2 ISI Interface chassis might have failed. Working with Dave and CDS to try to get someone out to try to replace it.

H1 General

Link

austin.jennings@LIGO.ORG - posted 08:02, Sunday 04 June 2023 (70113)

Ops Day Shift Start

TITLE: 06/04 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 9mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

- H1 is currently relocking at PREP_TR_CARM after almost a 24 hour long lock (so close!)

- SEI/CDS/DMs ok

H1 General

Link

anthony.sanchez@LIGO.ORG - posted 07:02, Sunday 04 June 2023 (70112)

Sunday Mid OWL Shift report

Current IFO Status: Unlocked
H1 just had a LockLoss after 23 hours and 41 minutes of NOMINAL_LOW_NOISE & OBSERVING.
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1369922215

It's currently unclear what cause the Lockloss.

H1 General

Link

anthony.sanchez@LIGO.ORG - posted 00:04, Sunday 04 June 2023 (70111)

Sunday OWL Shift Start

TITLE: 06/04 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 136Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:
Current IFO Status: NOMINAL_LOW_NOISE & OBSERVING
H1 has been locked for over 16 hours!

LHO General

Link

ryan.short@LIGO.ORG - posted 00:00, Sunday 04 June 2023 (70110)

Ops Eve Shift Summary

TITLE: 06/04 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 136Mpc
SHIFT SUMMARY:

H1 has now been locked for almost 17 hours. Other than a few small earthquakes passing through, very quiet shift. Wind and seismic are low.

ETMX saturations at 23:22, 23:32, 01:30, 02:25 (included a large DCPD saturation).

Handing off to Tony for the overnight.

LOG:

No log for this shift.

LHO General

Link

ryan.short@LIGO.ORG - posted 20:11, Saturday 03 June 2023 (70109)

Ops Eve Mid Shift Update

State of H1: Observing at 136Mpc

H1 has been locked and observing for 13 hours. A few small earthquakes rolling through and one large ETMX and DCPD saturation at 02:25 UTC, otherwise a quiet evening.

LHO General

Link

ryan.short@LIGO.ORG - posted 16:03, Saturday 03 June 2023 (70108)

Ops Eve Shift Start

TITLE: 06/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 134Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 7mph Gusts, 5mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY: Taking over from Austin. H1 has been locked and Observing for over 8.5 hours.

All systems okay
Dust monitors working and levels okay

H1 General

Link

austin.jennings@LIGO.ORG - posted 16:02, Saturday 03 June 2023 (70105)

Saturday Operator Summary

TITLE: 06/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 134Mpc
SHIFT SUMMARY:

- Arrived with the IFO locked and in OBSERVING

- Passing off to Ryan (in what is probably the shortest ops log I've ever written) with the IFO still locked and in OBSERVING - H1 is stable and SEI is low

- Lock #1:

Acquired NLN @ 14:15, OBSERVING @ 14:27
EX saturation @ 20:12

LOG:

No log for this shift.

H1 General

Link

austin.jennings@LIGO.ORG - posted 11:57, Saturday 03 June 2023 (70107)

Mid Shift Report

H1 has been locked for a 4:41 stretch, all systems seem stable.

LHO VE

Link

david.barker@LIGO.ORG - posted 10:11, Saturday 03 June 2023 (70106)

Sat CP1 Fill

Sat Jun 03 10:07:56 2023 INFO: Fill completed in 7min 55secs

Images attached to this report

H1 General

Link

anthony.sanchez@LIGO.ORG - posted 08:16, Saturday 03 June 2023 (70104)

Saturday OWL Shift End

TITLE: 06/03 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 132Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

7:37 UTC incoming 5.9M earthquake from Yemen.
8:26 UTC Seismic to calm.
8:33 UTC lockloss mostlikely due to ground motion from the many earthquakes that have been happening.
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1369816398

CAMERA_SERVO has been stalled since 9:56 UTC and is what is stopping us from gettign to Observing.
I tried manually moving the CAMERA_SERVO Guadian up to the next step but it was not moving up in states.
I Really wanted to just click down and rerequest the proper state, but I was unsure if it would cause a lockloss or not then force me to wait another hour to get back up to NLN again.
Contacted Keita at 10:30 UTC to see if he can help me understand more about what was going on, and determine if I can just click down and then CAMER_SERVO_ON again.
Keita pointed out the ASC Dither OverView screen which has the Camera scopes (Screenshots), and then it was obvious that the cameras were not working properly.
We then stepped through the CAMERA_SERVO Guardian code line by line to determine if we can take the CAMERA_SERVO Guardian Down without a lockloss.
Turns out the right thing to do was take CAMERA_SERVO Guardian to down then back up to CAMERA_SERVO_ON. We were successfully able to get to CAMERA_SERVO_ON NLN and get to OBSERING.

12:07 Lockloss possibly due to earthquake: https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1369829200

Relocking I had to use green arms Manual again tonight to lock Y arm.
Lost lock at TRANSITION_DRMI-TO-3F https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1369830883

Another lockloss this time from TRANSITION_DRMI-TO-3F https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1369831573

Current IFO Status: NOMINAL_LOW_NOISE & OBSERVING

Images attached to this report

H1 General

Link

austin.jennings@LIGO.ORG - posted 08:02, Saturday 03 June 2023 (70103)

Ops Day Shift Start

TITLE: 06/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 133Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

- IFO has been locked for 0:46 minutes

- SEI/DM/CDS ok

H1 General

Link

anthony.sanchez@LIGO.ORG - posted 05:08, Saturday 03 June 2023 (70102)

Saturday OWL Mid Shift Report

7:37 UTC incoming 5.9M earthquake from Yemen.
8:26 UTC Seismic to calm.

8:33 UTC lockloss mostlikely due to ground motion from the many earthquakes that have been happening.
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1369816398

9:55 UTC NLN was reached.
CAMERA_SERVO has been stalled since 9:56 UTC and is what is stopping us from gettign to Observing.
I tried manually moving the CAMERA_SERVO Guadian up to the next step but it was not moving up in states.
I Really wanted to just click down and rerequest the proper state, but I was unsure if it would cause a lockloss or not then force me to wait another hour to get back up to NLN again.
Contacted Keita at 10:30 UTC to see if he can help me understand more about what was going on, and determine if I can just click down and then CAMER_SERVO_ON again.

Keita pointed out the ASC Dither OverView screen which has the Camera scopes (Screenshots), and then it was obvious that the cameras were not working properly.
We then stepped through the CAMERA_SERVO Guardian code line by line to determine if we can take the CAMERA_SERVO Guardian Down without a lockloss. Turns out the right thing to do was take CAMERA_SERVO Guardian to down then back up to CAMERA_SERVO_ON.

Images attached to this report

H1 General

Link

anthony.sanchez@LIGO.ORG - posted 00:14, Saturday 03 June 2023 (70101)

Saturday OWL Shift Start

TITLE: 06/03 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 134Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 4mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

IFO Current Status: NOMINAL_LOW_NOISE & OBSERVING
VIolins are looking good.

LHO General (PSL)

Link

ryan.short@LIGO.ORG - posted 23:59, Friday 02 June 2023 (70099)

Ops Eve Shift Summary

TITLE: 06/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 133Mpc
SHIFT SUMMARY:

Lock #1:

PI mode 29 rung up at 23:58 (1:40 into the lock stretch); RMS reached ~2 but the guardian was able to damp it down successfully after about a minute of stepping through phases; took around another minute for the RMS to return to low level
Lockloss @ 4:23 (6 hours into lock stretch). Cause not obvious, however, low frequency LSC (ETMX) movement started 3 seconds before lockloss seen by several ASC and LSC signals including LSC MICH, SRCL, DARM, and DHARD P and Y
Camilla had noticed the ISS diffracted power had been trending upwards over the past several days, so after the lockloss I adjusted the Ref Signal to bring it down from around 3.0% to 2.2% (tagging PSL). This should keep our NLN input power closer to the desired 75W

Lock #2:

Lock acquisition went unaided. Both arms ran through INCREASE_FLASHES and DRMI caught quickly. NLN reached @ 5:21, Observing @ 5:26

Handing off to Tony.

LOG:

Start Time	System	Name	Location	Lazer_Haz	Task	Time End
23:19	VAC	Janos	MX	-	Dismantle and move pump	23:49

LHO General

Link

ryan.short@LIGO.ORG - posted 19:59, Friday 02 June 2023 (70100)

Ops Eve Mid Shift Update

State of H1: Observing at 132Mpc

H1 has been locked and Observing for over 4.5 hours. Rode out a 5.5 earthquake from Peru about a half hour ago, otherwise a quiet evening.

LHO General

Link

ryan.short@LIGO.ORG - posted 16:05, Friday 02 June 2023 (70098)

Ops Eve Shift Start

TITLE: 06/02 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 130Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 8mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY: Taking over from Austin. H1 has been locked for 45 minutes and Observing for 30 minutes.

CDS, SEI, SUS all okay
Dust monitors all working and levels okay
No alarms