aLIGO LHO Logbook

H1 General

Link

cheryl.vorvick@LIGO.ORG - posted 03:27, Sunday 05 March 2017 - last comment - 19:18, Sunday 05 March 2017(34584)

lockloss, temperature in PSL drops 2degF

waiting for temperatures to come back up
picture of refcav trans is attached
very off center, worse than last night, worse than 3 hours ago when I started my shift

Images attached to this report

Comments related to this report

cheryl.vorvick@LIGO.ORG - 03:45, Sunday 05 March 2017 (34585)

Link

WFS A DC PIT - good, then temperature drops, seems to be recovering, lockloss
PSL temperature drops 2degF in 14 minutes
EOM power drops, Periscope power drops, PZTs move by 100 in pitch and 40 in yaw

Images attached to this comment

peter.king@LIGO.ORG - 19:18, Sunday 05 March 2017 (34601)

Link

The picture is more likely to be the pre-modecleaner reflection, not the reference cavity transmission.
If it really is the reference cavity transmission, then we have a big alignment and mode matching
problem.

    The picture seems to indicate that the horizontal alignment has changed.

H1 DAQ (DAQ)

Link

cheryl.vorvick@LIGO.ORG - posted 01:13, Sunday 05 March 2017 (34583)

Verbal Alarms - timing system error - it's unclear what is expected when we get one of these

Verbal Alarms log looks like this:

Intention Bit: Undisturbed (Mar 5 05:47:27 UTC)
Timing system error (Mar 5 05:58:31 UTC)
Current time: (Mar 5 06:00:00 UTC)
Gamma Ray Burst (Mar 5 06:09:55 UTC)
Gamma Ray Burst
Acknowledge? (Mar 5 06:09:55 UTC)
Gamma Ray Burst (Mar 5 06:10:00 UTC)
Gamma Ray Burst
Acknowledge? (Mar 5 06:10:00 UTC)

Acknowledged (Mar 5 06:10:00 UTC)
ISC_LOCK state: LOCKLOSS (Mar 5 06:58:02 UTC)
DRMI Unlocked (Mar 5 06:58:02 UTC)
Intention Bit: Commissioning (Mar 5 06:58:06 UTC)

I looked up where this is in the VerbalAlarms code, and it's the channel H1:SYS-TIMING_ERROR_FLAG.

Activities to address:

Jim emailed DaveB, didn't hear back
I searched the wiki, both Ops and CDS, no guidance found
I looked at the VerbalAlarms wiki page, and then it's edits, but didn't find guidance
searched the alog and found only one reference, alog 23938
- "There is a timing error flagged on H1SUSETMY. Will wait for the next time IFO is out of Observing mode to reset it."
I posted the channel listed in alog 34581
- channels like H1:DAQ-BCST0_FRAME_WRITE_SEC that seem to be checking for if the data is good or not, but not connected

H1 General

Link

jim.warner@LIGO.ORG - posted 00:23, Sunday 05 March 2017 (34582)

Shift Summary

TITLE: 03/05 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 0Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG:

When I got here, Jeff and I noticed that the MC Refl spot was moving a lot. This seems to be associated with temps in the PSL. Kind of like the PSL AC is still on. Not sure if this has been causing lock losses.

Had two locklosses tonight, but relocking has been easy so far.

There also seems to be a timing issue, it's unclear to me if I should be waking people up to address this or not. Cheryl is looking.

H1 DAQ (DAQ, DetChar)

Link

cheryl.vorvick@LIGO.ORG - posted 00:11, Sunday 05 March 2017 (34581)

EDCU is red, 57 channels disconnected

daqd>
33776 channels
33719 connected
57 disconnected
33955 connection events processed
4136492044 value change events processed
H1:DAQ-BCST0_CYCLE
H1:DAQ-BCST0_TOTAL_CHANS
H1:DAQ-BCST0_DATA_RATE
H1:DAQ-BCST0_EDCU_CHANS
H1:DAQ-BCST0_EDCU_CONN_CHANS
H1:DAQ-BCST0_UPTIME_SECONDS
H1:DAQ-BCST0_LOOKBACK_RAM
H1:DAQ-BCST0_LOOKBACK_FULL
H1:DAQ-BCST0_LOOKBACK_DIR
H1:DAQ-BCST0_LOOKBACK_STREND
H1:DAQ-BCST0_LOOKBACK_STREND_DIR
H1:DAQ-BCST0_LOOKBACK_MTREND
H1:DAQ-BCST0_LOOKBACK_MTREND_DIR
H1:DAQ-BCST0_FAST_DATA_CRC
H1:DAQ-BCST0_FAULT
H1:DAQ-BCST0_BCAST_RETR
H1:DAQ-BCST0_BCAST_FAILED_RETR
H1:DAQ-BCST0_GPS
H1:DAQ-BCST0_CHANS_SAVED
H1:DAQ-BCST0_FRAME_SIZE
H1:DAQ-BCST0_SCIENCE_FRAME_SIZE
H1:DAQ-BCST0_SCIENCE_TOTAL_CHANS
H1:DAQ-BCST0_SCIENCE_CHANS_SAVED
H1:DAQ-BCST0_FRAME_WRITE_SEC
H1:DAQ-BCST0_SCIENCE_FRAME_WRITE_SEC
H1:DAQ-BCST0_SECOND_FRAME_WRITE_SEC
H1:DAQ-BCST0_MINUTE_FRAME_WRITE_SEC
H1:DAQ-BCST0_SECOND_FRAME_SIZE
H1:DAQ-BCST0_MINUTE_FRAME_SIZE
H1:DAQ-BCST0_RETRANSMIT_TOTAL
H1:DAQ-BCST0_PRDCR_TIME_FULL_MEAN_MS
H1:DAQ-BCST0_PRDCR_TIME_FULL_MIN_MS
H1:DAQ-BCST0_PRDCR_TIME_FULL_MAX_MS
H1:DAQ-BCST0_PRDCR_TIME_RECV_MEAN_MS
H1:DAQ-BCST0_PRDCR_TIME_RECV_MIN_MS
H1:DAQ-BCST0_PRDCR_TIME_RECV_MAX_MS
H1:DAQ-BCST0_PRDCR_CRC_TIME_FULL_MEAN_MS
H1:DAQ-BCST0_PRDCR_CRC_TIME_FULL_MIN_MS
H1:DAQ-BCST0_PRDCR_CRC_TIME_FULL_MAX_MS
H1:DAQ-BCST0_PRDCR_CRC_TIME_CRC_MEAN_MS
H1:DAQ-BCST0_PRDCR_CRC_TIME_CRC_MIN_MS
H1:DAQ-BCST0_PRDCR_CRC_TIME_CRC_MAX_MS
H1:DAQ-BCST0_PRDCR_CRC_TIME_XFER_MEAN_MS
H1:DAQ-BCST0_PRDCR_CRC_TIME_XFER_MIN_MS
H1:DAQ-BCST0_PRDCR_CRC_TIME_XFER_MAX_MS
H1:DAQ-BCST0_PROFILER_FREE_SEGMENTS_MAIN_BUF
H1:DAQ-BCST0_RAW_FW_STATE
H1:DAQ-BCST0_RAW_FW_DATA_STATE
H1:DAQ-BCST0_RAW_FW_DATA_SEC
H1:DAQ-BCST0_SCIENCE_FW_STATE
H1:DAQ-BCST0_SCIENCE_FW_DATA_STATE
H1:DAQ-BCST0_SCIENCE_FW_DATA_SEC
H1:DAQ-BCST0_STREND_FW_STATE
H1:DAQ-BCST0_MTREND_FW_STATE
H1:DAQ-BCST0_FRAME_CHECK_SUM_TRUNC
H1:DAQ-BCST0_SECOND_FRAME_CHECK_SUM_TRUNC
H1:DAQ-BCST0_MINUTE_FRAME_CHECK_SUM_TRUNC
Total 57 disconnected.
daqd>

H1 CDS (DetChar, OpsInfo)

Link

jim.warner@LIGO.ORG - posted 23:57, Saturday 04 March 2017 (34580)

Timing system in Error?

In addition to the frequent locklosses tonight, it seems that there is also a timing error. I've just noticed that GWIstat shows LHO as "TOO OLD", & DMT-Omega and the GDS range FOM have both stopped updating. I'm not sure what to do about these, but I'm sending an email to Dave.

H1 General

Link

jeffrey.bartlett@LIGO.ORG - posted 16:05, Saturday 04 March 2017 (34578)

Ops Day Shift Summary

Ops Shift Log: 03/04/2017, Day Shift 16:00 – 00:00 (08:00 - 16:00) Time - UTC (PT)

State of H1: Locked at NLN, range is 65.9 Mpc

Intent Bit: Observing for 18 hours.

Support: N/A

Incoming Operator: Jim

Shift Summary: Diag Reset for Timing error on H1IOPASCO. Ran A2L script – Pitch is slightly elevated and Yaw remains below the reference. Good shift for observing. No problems or issues to report.

Activity Log: Time - UTC (PT)

16:00 (08:00) Take over from Cheryl

16:17 (08:17) Diag Reset of H1IOPASCO Timing error

16:30 (08:30) Ran A2L check script

00:00 (16:00) Turn over to Jim

H1 General

Link

jeffrey.bartlett@LIGO.ORG - posted 12:13, Saturday 04 March 2017 (34577)

Ops Day Mid-Shift Summary

   At 14 hours into the current lock stretch, all looks good. Range has been around 66 to 67 MPC. The 0.1-0.3 Hz microseism has flattened out and may be starting to decline; primary microseism remains low. No concerns or problems to report from LHO at this time.

H1 FMP

Link

jeffrey.bartlett@LIGO.ORG - posted 08:53, Saturday 04 March 2017 (34576)

Alarm Handler Conn Error Mid-X

Alarm Handler - FMCS communications problem with the Mid-X AIRHANDLERS and TEMPERATURES.

H1 General

Link

jeffrey.bartlett@LIGO.ORG - posted 08:40, Saturday 04 March 2017 (34575)

ops Day Shift Transition

Ops Shift Transition: 03/04/2017, Day Shift 16:00 – 00:00 (08:00 - 16:00) - UTC (PT)

State of H1: IFO locked at NLN, at 29.4W and 65.7 Mpc

Intent Bit: Observing

Weather: Wind is up to a Light Breeze, Partly overcast, lower 40s

Primary 0.03 – 0.1Hz: Around 0.1um/s

Secondary 0.1 – 0.3Hz: At 0.4um/s and trending upward

Quick Summary: In Observing for 10.0 hours. No apparent problems or concerns

Outgoing Operator: Cheryl

H1 General (PSL)

Link

cheryl.vorvick@LIGO.ORG - posted 08:04, Saturday 04 March 2017 (34574)

Ops Owl Shift Summary:

TITLE: 03/04 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY:

drop in PSL temperature, noise seen in H1
LVEA also had one verbal alarm about low building temperature
otherwise quiet

H1 AOS (DetChar, PSL)

Link

cheryl.vorvick@LIGO.ORG - posted 05:53, Saturday 04 March 2017 - last comment - 06:34, Saturday 04 March 2017(34571)

Temperature in PSL drops - noise on PMC Trans, MC2 Trans

Not sure it killed the last lock, but for sure it caused noise that the IMC sees, both times the temperature has dropped in the last 24 hours.

Images attached to this report

Comments related to this report

cheryl.vorvick@LIGO.ORG - 06:17, Saturday 04 March 2017 (34572)

Link

Noise is clearly visible on the HAM2 ISS second loop qpd, and the X arm Trans A qpd.
Also, 11 hours ago when this happened, there was a realitively small increase in noise, and it seemed to recover, and then the noise got worse again, and H1 lost lock.
In this lock the small increase in noise has come and gone, and H1 just glittched in range down to about 10Mpc.

Images attached to this comment

cheryl.vorvick@LIGO.ORG - 06:34, Saturday 04 March 2017 (34573)

Link

taking a closer look at REFL_DC and the temperature, REFL recovers part or all of the way after the bursts of noise, at least in a time series, so lock loss may not be related

Images attached to this comment

H1 General

Link

cheryl.vorvick@LIGO.ORG - posted 00:30, Saturday 04 March 2017 (34570)

Ops Owl Transition:

TITLE: 03/04 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 66Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    Wind: 10mph Gusts, 6mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.38 μm/s
QUICK SUMMARY: both H1 and L1 have been locked about 2.5 hours

H1 General

Link

jim.warner@LIGO.ORG - posted 00:09, Saturday 04 March 2017 (34569)

Shift Summary

TITLE: 03/04 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 59Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG:
Survived a pretty good earthquake at 19:30, then lost lock after the ground had calmed down. I looked at the "stock" lockloss plots and it looked suspiciously like Travis' lockloss earlier today.

While trying to relock, I had problems with OMC locking. The first time I got to DC readout transition, it took 3 tries to get the OMC locked. I waited to check PI, violins etc, and immediately lost lock at DC readout. The second time, it again took 3 tries to get the OMC locked, but I was able to move on.

After getting to NLN, PI mode 27 rang up, but was unresponsive to phase changes. I tweaked the gain down to 5000 (don't know if this helped). Something got touched on mode 19 (not by me) though, and this is what initially dropped LHO out of Observe. I didn't have time to record what was changed, something PLL. Mode 27 eventually calmed down on it's own.

Images attached to this report

H1 SEI (ISC)

Link

jim.warner@LIGO.ORG - posted 20:17, Friday 03 March 2017 (34568)

Earthquake report

5.9 & 5.1 in Papua New Guinea

Was it reported by Terramon, USGS, SEISMON? Yes, Yes, No (Seismon still reports a 5.1 in Phek, India)

Magnitude (according to Terramon, USGS, SEISMON):5.9 & 5.1, 5.9 & 5.1, na

Location: Iceland region; 7.352°S 155.675°E

Starting time of event (ie. when BLRMS started to increase on DMT on the wall): 19:40 UTC, started seeing effects on IFO signals about 19:50.

Lock status? H1 stayed locked, range took a beating though. LLO is down

EQ reported by Terramon BEFORE it actually arrived? Unsure

Eq band BLRMS went up to ~1.5 micron for this eq, and we stayed locked. During the worst motion, PI mode 23 also "rang up" and down on it's own, maybe a coincidence.

H1 General

Link

travis.sadecki@LIGO.ORG - posted 16:00, Friday 03 March 2017 (34567)

Ops Day Shift Summary

TITLE: 03/03 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 66Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: One lockloss that had no readily apparent cause. Wind was in the high 30s for the first half of the day, but has calmed down to the high 10s for the remainder of the shift. IA was ran after lockloss, and no issues relocking.
LOG:

18:05 Betsy and Jason to MY

19:45 Betsy and Jason back

20:00 Lockloss

20:46 Richard running fiber in LVEA during IA

21:40 Observing

23:34 Out of Observe to run A2L before LLO comes back up

23:39 Back to Observing

H1 General

Link

travis.sadecki@LIGO.ORG - posted 13:41, Friday 03 March 2017 (34566)

Observing at 21:40 UTC

Did an initial alignment and had to damp some bounce and roll modes, but nothing out of the ordinary.

H1 PSL

Link

jeffrey.bartlett@LIGO.ORG - posted 06:30, Thursday 02 March 2017 - last comment - 18:41, Saturday 04 March 2017(34526)

PSL Reporting Low Temp

  At 14:19 (06:19) received alarm that PSL A/C was on or Low Temp in Laser Room. The PSL South temp is 69f, the PSL North temp is 70F. The PSL Table North is 73f. The PSL Table South is 71F. The Dust Monitor #101 under the table is reporting 69f.

Comments related to this report

jason.oberling@LIGO.ORG - 10:48, Thursday 02 March 2017 (34538)

Link

Here's a couple of trends around the relevant time. The first attachment shows all 4 PSL Enclosure laser room temperature sensors. The channels labeled *_ACS_* and *_ACN_* are the sensors at the output of the enclosure AC units, which are turned off during normal operation; the channels labeled *_TBLN_* and *_TBLS_* are the sensors attached to either end of the PSL table. The second attachment shows the south AC sensor (*_ACS_*) and the TPD signal from the FSS RefCav; I used the RefCav TPD as this has shown to be one of the more temperature sensitive alignment in the enclosure. As can be seen, as soon as the temperature begins to dip, the TPD signal becomes more jagged and begins to drop. This is consistent with a real change in temperature in the PSL enclosure. We are currently mystified by this behavior, as the AC units were both off during this time.

Similar behavior was reported by Travis yesterday, and again both AC units were off during the temperature dip. At this time we have no explanation for this behavior. Investigations continue and will update as we know more.

Images attached to this comment

john.worden@LIGO.ORG - 14:42, Thursday 02 March 2017 (34549)

Link

There is a fan in the LVEA return air plenum which continually pushes air into the PSL enclosure even when the AC units are off. This fan is behind the east wall next to the PSL. It would be worthwhile to find out what the temperatures are doing in that space. This return air space is much larger than the supply ducting to the LVEA and therefore the flow velocities are low and there may be air currents driven by convection. Also, winds will have the effect of pressurizing one side of our building with respect to the other side - this may also drive air currents in the return air plenum. Is there a correlation with wind conditions?

I do not know of any temperature sensors in the plenum.

jim.warner@LIGO.ORG - 18:41, Saturday 04 March 2017 (34579)

Link

It looks like something was changed on March 1st at about 20:00 UTC. PSL-ENV temp trends show a marked difference in behavior since then, see attached trends. It would be good to figure out what has happened because the temperature dips correlate with extra motion in the IMC, and we have lock losses that follow the temperature dips. Maybe not causal, but it looks like something is not nominal.

Images attached to this comment