We have had a timing at EY, which in turn caused a Dolphin glitch stopping the SUS, SEI and ISC models.
Dmesg on all three machines shows ADC timeouts. All IO Chassis are visible, so it looks like we just need to restart all the models.
I've opened FRS13623
To document the system before I started recovery, I captured all the front end system and Dolphin logs.
I then restarted the models in the order h1susey, h1seiey, h1iscey.
Finally did a global DIAG_RESET and cleared the DAQ CRC errors. All looks nominal.
I've added front end log details to the FRS (they suggest a gradual degradation of EY over a period of 8 minutes, initiated by h1seiey). I've marked the ticket with 3 hours observation time lost, and closed it as resolved (restarting models restored H1).
At the moment there is no obvious cause for the glitch, there was no EY activity at the time.
After 49+hrs of lock, H1 went DOWN with what appears to be an issue at EY (see frontends noted in Title & screenshot of Ops & CDS overview).
Over 47.5 hrs of lock for H1, and with a quiet night which keeps getting quiet as the winds die down.
HAM6 RGA assembly -> to be installed during HAM6 vent in October
Sheila, Daniel, Corey
Corey pointed out that the squeezer manager is giving a warning about the green power stabilization being railed. Over the last week Daniel and I have bumped up the green power (using the half wave plate) a couple of times, so that we are now using 24.3mW of power into the green fiber.
The transmitted power through the OPO is servo'd, so the launched power is increased to compensate. The OPO reflected power has also been increasing, which would suggest that the mode matching or alignment into the OPO has increased, or that the losses are changed.
For now the operators can ignore the message about the green power stabilization, we may want to lower the green power setting for the OPO for the next week. Daniel is planning to look at a mode scan of the OPO on Tuesday to see if there is evidence that the mode matching or alignment has degraded.
TITLE: 09/20 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 19mph Gusts, 16mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.18 μm/s
All looks fine & good seismically and wind-wise.
QUICK SUMMARY:
H1 made it back to Observing in the last hour and continues with a ~44hr lock (!).
TITLE: 09/20 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Observing at 110Mpc INCOMING OPERATOR: Corey SHIFT SUMMARY: Remained locked entire shift. Dropped out of observing a couple of times for PEM injections and measurements. LOG: 15:03 - 15:16 UTC Niko to optics lab H0:FMC-MX_AH_COOLTEMP_2_DEGF invalid alarm 16:09 UTC H0:FMC-MX_AH_COOLTEMP_1_DEGF invalid alarm H0:FMC-MX_AH_DSCHRG_DEGF invalid alarm 17:20 - 18:25 UTC Dropped out of observing for PEM injections. Robert and team to LVEA. 17:30 - 18:03 UTC Tyler moving bee nest with forklift 18:01 - 18:20 UTC Chandra to mid X 18:56 UTC Dripta to optics lab 19:07 UTC Ethan to optics lab 19:30 UTC Ethan and Dripta out of optics lab 20:20 - 20:33 UTC Gerardo to mid X 21:04 UTC Switched to commissioning for Sheila to run measurements 21:34 - 21:55 UTC Robert and team to LVEA to set up accelerometers 22:26 UTC Robert to LVEA 22:31 - 22:57 UTC Kyle to mid stations 22:32 UTC Back to observing 22:52 - 23:02 UTC Dripta to optics lab
Sheila, Rahul
We have been trying to follow and analyze the CARM_TO_TR locklosses. In the last week or two there have been at least 8-10 locklosses attributed to this. Based on the list of channels created by Sheila for diagnostics, I am posting the last 6 CARM_TO_TR locklosses. Looking at the channel (QPDs at the end of both the arms, sums of which are used for monitoring the arm powers) H1:ASC-X_TR_B-NSUM_OUT_DQ, its ringing up with frequencies ranging from 60-100Hz. I will have a look at it during different stretches of lock.
The time zone is local. The down arrow opens up a calendar. Links to the Python script at /opt/rtcds/userapps/release/sus/h1/scripts/restore_suspensions.py. Uses nds. Screenshot attached.
TITLE: 09/20 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 114Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 6mph Gusts, 5mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.17 μm/s
QUICK SUMMARY:
Can not connect to TeamSpeak.
H0:FMC-MX_AH_COOLTEMP_2_DEGF is invalid.
H1:GRD-SQZ_MANAGER_USERMSG: green power stabilization railed
Robert has PEM injections scheduled for 10 AM.
Carlos fixed TeamSpeak connection.
PEM injections moved to 10:20 AM.
TITLE: 09/20 Owl Shift 07:00 – 15:00 (00:00-08:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Quiet shift. Locked for just under 36 hours.
LOG:
07:00 (00:00) Start of shift
09:54 (02:54) Transitioning to EQ mode for 5.3 near Tonga
10:11 (03:11) EQ rung down, transitioning back to windy
15:00 (08:00) End of shift
Ops Shift Transition: 09/20/2019, Owl Shift 07:00 – 15:00 (00:00-08:00) - UTC (PT)
State of H1: Locked
Intent Bit: Oberving
Weather: 0-5 mph wind
Primary 0.03 – 0.1Hz: 0.01 um/s
Secondary 0.1 – 0.3Hz: 0.2 um/s
Outgoing Operator: Travis
Quick Summary: Locked for a little under 28 hours.
TITLE: 09/20 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Niko
SHIFT SUMMARY: Locked the entire shift. One smallish EQ that we rode through without issue.
LOG:
23:15 Commissioning mode for PEM injections
0:18 Observing. PEM crew noted that they heard several "whoomp" noises while in the LVEA. They are speculating that it is possibly due to wind, although wind was low at the time.
5:46 Transitioned to EQ mode.
6:47 Back to WINDY mode.
TITLE: 09/19 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 11mph Gusts, 6mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.14 μm/s
QUICK SUMMARY: Lock is 20 hours old. PEM injections commencing soon.
TITLE: 09/19 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Travis
SHIFT SUMMARY: Locked for 19.5 hours, no issues.
LOG:
As part of the point absorber paper, I've been looking extremely closely at the ITMY point absorber distribution. Since the heat from the points tends to diffuse into the bulk over time, taking a measurement over a very short time scale yields a much more distinct view of the individual point absorbers. By looking approximately 120s after a power-up, I was able to isolate at least 6 distinct point absorbers on ITMY.
| Point | X coord [mm] | Y coord [mm] |
| 1 | 0 | 0 |
| 2 | -18 | -17 |
| 3 | -15 | +48 |
| 4 | -52 | -25 |
| 5 | +30 | -8 |
| 6 | +5 | -35 |
I've centered the HWS coordinate system on the biggest point.

The distribution is such that after 20 minutes, the individual points are much less distinct.

TITLE: 09/19 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 3mph Gusts, 1mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.18 μm/s
QUICK SUMMARY: 11.5 hour lock, calm environment.
TITLE: 09/19 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 112Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: locked all shift
LOG: