Looks like the CAMERA_SERVO had a CONNECTION ERROR for h1digivideo3 from 13:21:20-13:22:00 (taken out of observing during the error and taken back to OBSERVING at 13:22:01utc).
h1digivideo0 ran out of memory at 06:20 PDT this morning and the ETMX video server process crashed. It was restarted by monit (via systemd), which recouped 24% available memory. Attached 6 hour trend of mem available.
TITLE: 08/27 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 3mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.07 μm/s
QUICK SUMMARY:
H1's been locked for almost 9hrs w/ a range of around 160Mpc.
On CDS Overview, there is a little RED for the SUSH2B front ends (attached, looks like CRC Sum errors).
Also happened to notice nuc30's H1 live DARM disappear for about 3-5 seconds at 1508utc.
TITLE: 08/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:
- Arrived to an unlocked IFO, ran into the same issue as last night but was able to reconcile with these steps
- Back to NLN @ 1:27/OBSERVE @ 1:51 UTC
- EX saturations @ 3:47/4:04/4:22/4:35
- Lockloss @ 5:06
- Reaquired NLN unaided @ 6:09/OBSERVE @ 6:25
- EX saturation @ 6:36
- Setting H1 to auto and passing to Tony for the night
LOG:
No log for this shift.
Lockloss @ 5:06 - a short 3.5 hour lock, possible ASC ringup? Saw the ASC signals start to oscillate with growing amplitude on nuc29 leading up to the lockloss - scope attached.
Had some issues relocking after the lockloss, similar to last night, but has since been resolved. H1 has been locked and observing as of 1:51 UTC. Ground motion/wind low.
TITLE: 08/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY:
Just as Austin and I were chatting at the hand-off, H1 lost lock! :(
LOG:
TITLE: 08/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 17mph Gusts, 14mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY:
- H1 has just lost lock as I am typing this...cause unknown
- SEI motion is a bit elevated in the EQ band, hovering around 0.1 um/s
Upon the lockloss, we had the same issue with ALS Y not catching any light. I used the same solution as Tony did this morning by restoring the ETMY/ITMY/TMSY sliders to the last time an initial alignment was completed, ~1376843785. After playing with the sliders for a bit once this was restored, I was able to get light on the PD. Being that this is the second time that we have had to restore sliders, perhaps this is evidence of some slow drift (possibly related to the chiller failure) going on at EY? Will investigate this further.
Update:
Was able to catch ALS Y and lock it, but had a few subsequent locklosses following LOCKING ALS. I will try running another initial alignment to see if this helps. ALS Y still had issues catching on IA.
I tried moving ITMY just a hair, flashes would get right below if not at 1 on the LOCKING green arms ndscope template, but when the WFS would turn on it would immediately lose lock. After a few trial and error attempts, I was able to reconcile this by turning OFF the WFS DOF 1/2 Y (found under the EY overview) immediately when they get turned on in ENABLE_WFS. Then, after 5 seconds or so, I would turn ON the WFS DOF 2 Y first, then WFS DOF 1 Y which then allowed the WFS to converge without breaking the lock.
Had to touch SRM by a bit in P for SRC to catch but otherwise the rest of the initial alignment went unaided.
Attached is a 24 hour trend of the witness sensors and slider values for ETM/TMSY and the oplev trends for ETMY. Looking at the scope, I'm not seeing a whole lot of drift of the course of the day, so I'm a bit perplexed why we ran into the same issue again this afternoon, since obviously there's some issue with the Y arm alignment.
Happened to look up and see a "Stand Down" state. Did not receive any alerts for this. The graceDB link for this lists this "GRB-SHORT" as NOTGRB. Here's more info on this particular event (E432741): https://gracedb.ligo.org/events/E432741/view/
PSL Report FAMIS report:
Laser Status:
NPRO output power is 1.829W (nominal ~2W)
AMP1 output power is 67.15W (nominal ~70W)
AMP2 output power is 135.5W (nominal 135-140W)
NPRO watchdog is GREEN
AMP1 watchdog is GREEN
AMP2 watchdog is GREEN
PMC:
It has been locked 20 days, 6 hr 1 minutes
Reflected power = 16.33W
Transmitted power = 109.2W
PowerSum = 125.5W
FSS:
It has been locked for 0 days 10 hr and 48 min
TPD[V] = 0.9056V
ISS:
The diffracted power is around 2.4%
Last saturation event was 0 days 10 hours and 48 minutes ago
Possible Issues: None
H1's been locked for 8.5hrs. At around 1630utc (930amPT) H1 took a step down in range. There was also a brief drop out of Observing due to the Squeezer (noted earlier).
At 17:41:44, H1 was dropped out of OBSERVING...after quickling taking OBSERVATORY_MODE to "COMMISSIONING", H1 was ready to go back to OBSERVING at 17:42:21.
Using the Guardian Control command, there was a brief SDF diff due to syscssqz. I am not sure how to find which channel dropped us out. I ran guardctrl on Squeezer nodes and a few other guardian nodes, but could not find the culprit. (see attached).
Sat Aug 26 10:10:48 2023 INFO: Fill completed in 10min 44secs
TITLE: 08/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 13mph Gusts, 11mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY:
H1's been locked 4hrs. H1 range continues at increased value of 150Mpc as was for the lend of the last lock. There was a Labs Dust alarm needing to be acknowledged (it had been OK for last3hrs). Winds slowly beginning to pick up.
Hanford site road work continues on a Saturday on Route10 (crack sealing) and I was turned around and had to take the long way around to get to work (this work was normally M-Th 7am-3pm, but they've been working Fri & Sat this week).
h1digivideo3's memory usage is getting high, we will keep an eye on it and perhaps restart a process during a target-of-opportunity.
TITLE: 08/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:
- EX saturation @ 11:10/1:48/2:13 UTC
- Lockloss @ 5:56 - new record for a O4 lock at LHO - 60:29!
- Issues getting light on the ALS Y PD - might be related to the chiller issue we had earlier today?
LOG:
No log for this shift.
ALSX-REFL_A_DC
Has no voltage and low power.
Running the Dither Align script on TMSX and ITMX in hopes that it would fix the issue with ALSX ReflPD A
It did not.
I still have an issue with ReflPD A And PDH.
To get the X are ReflPD A issue resolved.
I had initially thought that is was something that wasn't getting power or voltage, but since I was able to close the shutter and see a change in H1:ALS-X_REFL_A_DC_POWER, I realized that everything is indeed working correctly, it's just very poorly aligned.
I tried, rolling back to a few hours ago before oplevs had moved about 4 hours prior.
I Tried baffle dithering twice, following both this https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=62024
and the Dithering instructions found here: https://cdswiki.ligo-wa.caltech.edu/wiki/Troubleshooting%20the%20IFO
This made the flashes I was seeing brighter, but it was still too low for the REFLPD A error 8 to be resolved.
I drove to site so I can get my eyes on the how everything was behaving, and so I can have more screen real estate to see some more trends.
Trending ISC_LOCK back to the last Initial_Alignment gave me a GPS time of 1376843785 or so. Once I got that I put all of the ETMX, ITMX & TMSX back to that location.
I then started an Initial alignment. Stopped Initial alignment are SRC align, and followed trhough walking through initial alignment
Locking Started at 10:46 UTC
Had to accept this SDF to get to observing.
Observing Reached !!!