H1 is down and in corrective maintanence to troubleshoot an issue with the PMC. Currently working with team PSL to diagnose the cause.
The timing system has a corner station error: the MSR atomic clock 1PPs difference went from -159nS to 0.11S at 19:31 PDT this evening.
This signal is for verification purposes and does not drive the timing system. We will take a look at what has happened to it in the morning.
The atomic clock was fairly stable over the past 50 days or so, and then jumped. Not sure what could cause this short of a power glitch.
LOCKLOSS @ 2:02, had a SRM saturation right before the lockloss. Seeing some movement in INP1 P and some LSC instability as well.
Looking at the WIND and the SRM, the wind seems to gusting between 10-17mph at the same frequency as the oscillations in the SRM starting at 1m,21s before lockloss.
Famis 18649
| Level (inches) | ||||
|---|---|---|---|---|
| Location | Value | Difference from last reported levels |
Drip Pans | Leaks |
| CS | 5 9/16 | -1/16 | Clean | No unaddressed puddles |
| EX | 7 1/4 | -1/16 | Clean | No unaddressed puddles |
| EY | 8 9/16 | 0 | Clean | No unaddressed puddles |
Famis Task: 23750
I wanted to just swap the pumps with the one below the table but couldn't reach the pump rebuild log https://dcc.ligo.org/LIGO-Q2200010 because i didn't have any internet at the end stations and couldn't determine if the spares had been rebuilt already or not.
EX
Dust pump number 2950 Not running and Unplugged.
I checked the Log and noticed that pump 2950 only lasted 2 months. But the pump below the table was newly built around the same time.
So I'm going to Leave the Famis Task open until I can swap out the pump.
EY
Dust pump was Running pressure was 19.5 inHg.
No temps over 151F .
Corner Station
Dust pump number 1194 Power not switched on.
Spare 1119313238 has also been rebuilt and is ready to be swapped out.
I will try to swap these out Tomorrow or Tuesday.
UPDATE!! CS Dust pump Pump Power not switched on. I spoke to TJ about it. It's on a breaker and I beleive the breaker tripped. I had to turn it off before I could turn it back on. When I turned it on It sputtered to live and puffed a rather large puff of dust out. So much so it startled our Surf student Rachel and I. It was getting vacuum pressure around -14 inHg. So I swapped it out for 3238. The Brass threaded fitting below the 4 way splitter that the gauge is afixed to in the top of the housing is stuck and I couldn't remove it for the swap. I ended up finding a good replacemet for the threaded brass fitting to go into the next pump i swapped it for. The new pump # ending in 3238 is running great and has more than enough vacuum pressure, I set the pressure to be at -19inHg. The pump that was removed was placed in the OSB receiving room to be rebuilt or otherwise just cleaned up. EX Dust pump Pump number ending in 2950 Not running Unplugged. I plugged it in and noticed that it was not getting very past -9 inHg and sounded unwell. I came back the next day to find that the pressure was still too low though it had creeped up to -13inHg. I then swapped out the pump with the one under the table Pump # ending in 2109which was newly rebuilt. This too was still too low of a pressure around -14.5inHg. I checked the filters for tightness on the front of the pump housing. I then jiggled the vacuum hose a little and saw a the pressure go back up. So I checked the hose fitting and noticed that the hose was affixed to a plastic fitting that has threads going into to the brass 4 way fitting without any plumbers tape. I then put some plumbers tape on it, put it back together and fired her back up. I still didn't see the pressure we desired it's only at -17inHg. I went back to day to check on it and it's still at 17inHg. There is not any more room for adjustment on the vacuum pressure adjuster. The old pump is in the OSB recieving room. I will try to open up those pumps next week.
TITLE: 06/11 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 136Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 13mph Gusts, 11mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.10 μm/s
QUICK SUMMARY:
Took over the IFO as it was relocking.
NOMINAL_LOW_NOISE reached at 15:38 UTC
Observing at 15:50 UTC
Lockloss 17:48 UTC
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=70342
Relocking: No initial Alignment done
NOMINAL_LOW_NOISE Reached 18:41 UTC
Observing 18:45 UTC
Lockloss 21:45 UTC
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=70343
Relocking : No Initial alignment.
NOMINAL_LOW_NOISE reached at 22:42 UTC
Observing at 22:54
TITLE: 06/11 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 137Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 11mph Gusts, 6mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.10 μm/s
QUICK SUMMARY:
- H1 has been locked and in observing as of 22:42
- CDS/SEI ok, SEI motion is a little high - will monitor
Lockloss most likely caused by ground motion:
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1370555148
Peakmon plots attached.
There was no Earthquake warning just a sudden Lockloss. The Seismic foms showed a sharp but small increase which pointed me in the direction of looking for Seismic info. There was a sudden increase in H1:ISI-GND_STS_ITMY_Z_BLRMS_30M_100M and other Z axis BLRMS.
I am surprised that the Lockloss Analysis tool didn't tag it as an Earthquake.
Lockloss URL:
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1370540915
The Lockloss ndscopes show that H1:ASC-CSOFT_P_OUT16 and H1:ASC-DHARD_P_OUT16 are elevated relative to the other plots on the screen, But trending those channels back an hour or even further in the lock shows the same behaivor during the middle of the lock. Also comparing those values to the live data of a current lock they all seem to be within a reasonable and tolerable range of motion for H1 in a NOMINAL_LOW_NOISE & Observing state.
The Lockloss URL shows some saturation plots that may point to ETMX L3 and to H1:SUS-ETMX_L3_MASTER_OUT.
Perhaps this is the actuation point fo ETMX and is just pointing out the obvious.
Cause is Undetermined.
Sun Jun 11 10:11:14 2023 INFO: Fill completed in 11min 13secs
TITLE: 06/11 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 6mph Gusts, 2mph 5min avg
Primary useism: 0.07 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY:
There is still some ground motion happening. be we are going to try to relock.
IFO Current status: Relocking due to earthquake.
TITLE: 06/11 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY: Multiple locklosses overnight from earthquakes.
Lock #1:
Lock #2:
Lock #3:
Lock #4;
Handing off to Tony for the day.
LOG:
No log for this shift.
Lockloss @ 14:14 UTC from a magnitude 5.8 earthquake from New Zealand. This lockloss happened at the same time as seismic configuration moved to EARTHQUAKE.
Holding in DOWN for ground motion to settle, then will start relocking.
Just a note that the lockloss tool did not log this as an earthquake lockloss. The buildups are very clearly oscillating at low frequency in the last few minutes leading up to the lockloss, and the seismic FOM shows increased ground motion. I am not sure what triggers an earthquake tag in the lockloss tool but I figured it would be useful to know that the tool did not properly classify this one.
Lockloss @ 10:05 UTC from what seems to be a nearby and sudden seismic event. The motion appeared quickly in both 0.03-0.1 and 0.1-0.3 Hz frequency bands and Picket Fence did show small motion on all sensors a minute or so after the lockloss. This event does not correspond to any earthquake reported by USGS.
Seismic configuration did not switch to EARTHQUAKE until the second wave hit (around -5m on attached ndscope). Waited in DOWN for about 10 minutes for ground motion to calm down, now relocking (and anticipating an incoming 6.2 from Japan).
The timing of this event in relation to the 6.2 magnitude quake from Japan is noteworthy, but this lockloss occurred before seismon or the USGS site reported the incoming Japan quake, which is why I hesitate to say whether the two events are the same or not (but still entirely possible).
TITLE: 06/11 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 135Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 9mph Gusts, 7mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY: Taking over from Austin. H1 has been locked and Observing for over 1.5 hours.
TITLE: 06/10 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 136Mpc
SHIFT SUMMARY:
- Lock #1:
- Lock #2:
- After the EQ, decided to do an IA just to make sure the alignment is still ok
- Lock #3:
- Leaving the IFO to Ryan S. locked and in observing
LOG:
No log for this shift.
H1 just unlocked potentially due to an 5.2 EQ from Japan (though I'm a bit skeptical that this was 100% the cause for reasons listed here). Doing an IA just to be safe and will begin relocking shortly.
J. Oberling, R. Savage, A. Jennings
Logged in from home, all signals looked OK, we had RF and high voltage to the PZT, signal on the locking PD, and the ramp worked. Tried locking and got nothing. There was no obvious reason why it wouldn't lock (same as Ryan and Jenne saw when Austin called them). Since I couldn't tell what the issue was from home, I drove to the site to take a look; Rick joined in case we needed to swap something out. Austin took a look at the electronics (PSL rack and CER), and I confirmed, all electronics visibly OK, nothing seemed out of place. Decided to watch the signal on the locking PD and turned on the PZT ramp. Flashes could be seen, but nothing down around the trigger threshold level, so I increased it from 0.1 V to 0.2V (where I had seen the threshold drop to a couple times), and the PMC locked right up without issue. First attachment shows when I changed the locking threshold and the resulting relock. Not sure why the threshold needed to be changed, but it's locked now. Unlocked and locked it several times, and it locked every time without issue. For some reason I currently do not know, the PMC would consistently not lock with a threshold of 0.1V, even if the locking PD voltage dropped below 0.1V; the 2nd attachment shows several instances when the PD voltage dropped below 0.1V, but the PMC did not lock (if the PMC locked then the PD voltage would have dropped to its usual locked value of ~20mV). This threshold value has been in place since at least the end of the 2021 PSL laser upgrade (likely back during O3 as well), so I'm not sure why the sudden change (and why I didn't think to try this first; lesson learned). I tried with the locking threshold at 0.15V, and this also locked without issue (3rd attachment), so I left the threshold here. I accepted the change in SDF; I didn't get a screenshot but did get a picture with my phone, will add that as a comment to this alog. At the time of this writing Austin has gone through an initial alignment and is now trying to lock the IFO; PMC so far has been stable, will monitor through this locking process.
FRS ticket 28260 has been created to track this issue.