TITLE: 06/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 133Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

- IFO has been locked for 0:46 minutes

- SEI/DM/CDS ok

7:37 UTC incoming 5.9M earthquake from Yemen. 
8:26 UTC Seismic to calm. 

8:33 UTC lockloss mostlikely due to ground motion from the many earthquakes that have been happening. 
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1369816398

9:55 UTC NLN was reached.
CAMERA_SERVO has been stalled since 9:56 UTC and is what is stopping us from gettign to Observing.
I tried manually moving the CAMERA_SERVO Guadian up to the next step but it was not moving up in states.
I Really wanted to just click down and rerequest the proper state, but I was unsure if it would cause a lockloss or not then force me to wait another hour to get back up to NLN again. 
Contacted Keita at 10:30 UTC to see if he can help me understand more about what was going on, and determine if I can just click down and then CAMER_SERVO_ON again.

Keita pointed out the ASC Dither OverView screen which has the Camera scopes (Screenshots), and then it was obvious that the cameras were not working properly. 
We then stepped through the CAMERA_SERVO Guardian code line by line to determine if we can take the CAMERA_SERVO Guardian Down without a lockloss. Turns out the right thing to do was take CAMERA_SERVO Guardian to down then back up to CAMERA_SERVO_ON. 

 

TITLE: 06/03 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 134Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 4mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

IFO Current Status: NOMINAL_LOW_NOISE & OBSERVING
VIolins are looking good.

TITLE: 06/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 133Mpc
SHIFT SUMMARY:

Lock #1:

• PI mode 29 rung up at 23:58 (1:40 into the lock stretch); RMS reached ~2 but the guardian was able to damp it down successfully after about a minute of stepping through phases; took around another minute for the RMS to return to low level
• Lockloss @ 4:23 (6 hours into lock stretch). Cause not obvious, however, low frequency LSC (ETMX) movement started 3 seconds before lockloss seen by several ASC and LSC signals including LSC MICH, SRCL, DARM, and DHARD P and Y
• Camilla had noticed the ISS diffracted power had been trending upwards over the past several days, so after the lockloss I adjusted the Ref Signal to bring it down from around 3.0% to 2.2% (tagging PSL). This should keep our NLN input power closer to the desired 75W

Lock #2:

• Lock acquisition went unaided. Both arms ran through INCREASE_FLASHES and DRMI caught quickly. NLN reached @ 5:21, Observing @ 5:26

Handing off to Tony.

LOG:                                                                                                                                                                                                                                                                                                

State of H1: Observing at 132Mpc

H1 has been locked and Observing for over 4.5 hours. Rode out a 5.5 earthquake from Peru about a half hour ago, otherwise a quiet evening.

TITLE: 06/02 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 130Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 8mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY: Taking over from Austin. H1 has been locked for 45 minutes and Observing for 30 minutes.

• CDS, SEI, SUS all okay
• Dust monitors all working and levels okay
• No alarms

TITLE: 06/02 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 134Mpc
SHIFT SUMMARY:

- Arrived with the IFO down due to ISS chassis failure, turned out to be a loose cable inside the chassis - alog

- After recovery, I began an initial alignment where we saw yet again SRC2 ASC turning on before the camera spot looked good, indicating that there is something wrong with one of the thresholds

• We took ALIGN IFO to DOWN to get some scopes up to see what was going on, but when rerequesting SRC ALIGN it magically decided to work (so this issue will still have to be looked at in the future)

- 1941 - Another alarm went off in the CR hallway, most likely the fire alarm? The sound did couple into DARM at the low-mid range frequencies. Turned off by itself after about a minute. This seems to be an ongoing thing - Tagging FAC

- IY 7 G: 0.2 is a new violin mode setting I have been testing and so far seems to be working great. I have not yet put this change in lscparams as I would like to continue to test this gain setting over the next few locks

• Some more settings from Ryan C for testing: IY 5/6 G = -0.04 (was -0.01), IY4 G = +0.4 (was +0.1), IY8 G = -0.2 (was -0.08)

- Lock #1:

• Acquired NLN @ 18:20, OBSERVING @ 18:29
• EX saturation @ 20:05
• PI 29 ringup @ 20:11, subsequent LOCKLOSS shortly after, couldn't damp it in time

- Lock #2:

• Struggled with ALS Y on relocking, neither I nor INCREASE FLASHES could get flashes above ~0.8
• Finally was able to get ALS Y to lock by walking ETMY/TMSY in Y in LOCKED_SLOW_WFS_ETM_TMS but lost lock shortly after, doing an IA
• Ran into the issue with SRM offloading bad values during this IA, so I took ALIGN IFO to SR2 OFFLOADED and tried again
• After a couple tries, it caught SRC ALIGN, but AS AIR look very poor so once in SRC ALIGN OFFLOADED, I adjusted SRM ~110 microradians in Y to get the peaks on AS A DC NSUM and the AS air camera to look better, which seemed to have worked
• Acquired NLN @ 22:17, OBSERVING @ 22:28

LOG:                                                                                                                                                                                                              

I've made a screen and that has the external candidate event info and standdown event info. On here is also some controls to cancel stand downs and make stand downs for times: 1 hour, 1 day, indefinite. This will illuminate a bright orange STAND DOWN on this screen, and the usual orange stand down light will be visible in the control room next to the lock clock. More of this info might migrate to the ops overview eventually.

This screen is availble from the OPS dropdown on the sitemap as "EXT ALERT STAND DOWN"

The IFO is locked and has been in OBSERVING for just over 1 hour.

Fri Jun 02 10:07:25 2023 INFO: Fill completed in 7min 25secs

Gerardo confirmed a good fill curbside.

J. Oberling, T. Shaffer, R. McCarthy, M. Pirello, A. Jennings

At approximately 8:56 UTC (1:56 PDT), the ISS first loop failed, causing a lockloss a few seconds later.  Tony called me shortly after 7:00am PDT, and I logged in remotely to take a look.  All ISS signals were reading incorrectly (ISS PDA and PDB were both reading <1V, AOM drive voltage was reading negative, ISS QPD very noisy; see attached trend), but we still had a running laser and a locked and operating PMC and FSS RefCav.  This is important to note as if we had lost any DC power (like one would first suspect upon seeing both PDs and the QPD "fail" at the same time) we would also not have an operating laser, a PMC, or a RefCav (the PMC/RefCav locking servos and PSL control box in the PSL LVEA rack all pull from the same +/-24V and +/-18V DC supplies, so losing one would take out all items connected to that supply; PSL CER rack uses a different +/-18V DC power supply).  Suspecting a failed AA chassis, since everything ISS-related was down, I drove to the site and upon arriving, sure enough, TJ and I found the ISS AA chassis was reporting no -15V.  Richard and I pulled the chassis and he found the -18V wire running from the ON/OFF rocker switch to the voltage regulator board (which takes the +/-18V to +/-15V) was very loose (it fell out of the crimp connector upon being touched).  Marc re-crimped the wire with a new connector and powered up the chassis; everything came up as expected.  We then installed it, and lo and behold we have ISS signals that make sense.  I walked Austin through re-engaging the ISS and everything came back to their normal operating points.  With the ISS now working correctly, Austin has begun relocking the IFO; at the time of this writing they are at "Find IR."

Serial number for this AA chassis is S1001249.

Edit to add: Clarification of +/-24V mention only applying to PSL rack items, not CER; added serial number of PSL ISS AA chassis.

I just started the well pump to add water to the fire water storage tank. The well pump will run for 4 hours and automatically turn off. 

TITLE: 06/02 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Aligning
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 7mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY:

IFO STATUS : UNLOCKED  with potential PSL Issue.
PSL-ISS Defracted power is still too high.

The Light on the H1:PSL-ISS_PDA_CALI_DC_OUTPUT has gone from 9V to less than 1V.

There are a number of PSL channels that all have a change in their signals around the same time. Both in the ISS Second loop and first loop.
Tracing back the from the Second loop, through the first loop points to something potentially right after the PMC. Jason has been informed and is heading to site to go inside the PSL and troubleshoot.

TITLE: 06/02 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 6mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s 
QUICK SUMMARY:

- IFO is DOWN due to current issues with the ISS, link to Tony's alog here, Jason is going to look into it

- CDS/SEI/DMs ok

IFO STATUS: UNLOCKED
Sudden Lockloss for an Unknown reason 34 minutes into lock: 
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1369731497

Tried to relock right after the lockloss and has a lockloss at LOCKING ALS

I'm having troubles getting through Align_IFO's INPUT_ALIGN_OFFLOADED because the IMC is losing lock during the process.
I'm now going to try pausing both INIT_ALIGN and ALIGN_IFO until the IMC is locked again.
Seems like the IMC is taking a while to get Locked.
Started troubleshooting Init_Align: 
https://cdswiki.ligo-wa.caltech.edu/wiki/Troubleshooting%20the%20IFO#Initial_Alignment_Troubleshooting

Got to the diagnosing IMC Locking Issues.
https://cdswiki.ligo-wa.caltech.edu/wiki/Diagnosing%20IMC%20locking%20issues

DiagMain has a message PSL_ISS: Diffracted power is high.
Searched for an Alog about this message.
Trending these channels: I noticed that these signals 
H1:PSL-ISS_DIFFRACTION_MIN H1:PSL-ISS_DIFFRACTION_AVG H1:PSL-ISS_DIFFRACTION_MAX

Using alog 58264 as a reference I followed the instructions for adjusting the diffraction power.
 H1:PSL-ISS_REFSIGNAL current value: -1.99 -> back to -1.99
 H1:PSL-ISSSECONDLOOP_AC_COUPLING_OFFSET Current Value 0.050 -> back to 0.050

command for the ndscopes I was using to determine if my changes are actually doing anything at all:
ndscope H1:PSL-ISS_SECONDLOOP_AC_COUPLING_OFFSET H1:PSL-ISS_DIFFRACTION_AVG  H1:PSL-ISS_SECONDLOOP_ENABLE_MON H1:PSL-ISS_REFSIGNAL H1:IMC-PWR_IN_OUT16 -k
It seems like no matter what I do with the REFSIGNAL there is not a responce on the Diffaction AVG even if I change the AC Coupling Offset. 
 
 H1:PSL-ISS_SECONDLOOP_CLOSED and Cylced this on and off. 

IMC can lock when ISC goes into Prep for locking. But still has a PSL-ISS: Diffraction Power too High error message. And IMC is still unlocking during Initial Alignment.

I have taken the opportunity of H1 being out of data taking mode to restart the redundant SenseMonitor processes. Note that these processes are not used for the dmtviewer or for summary page generation. The reason for this restart is that the redundant trend frames had the same names as the the primary ones and they were interfering with the gw_data_find tracking of the primary frames. The primary frames are not affected, but the redundant frames are now named, e.g. SenseMonitor_hoft_H1_R_M.

TITLE: 06/02 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 4mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:
IFO Current status: Locking arms Green.
OMC_DCPD lock loss?
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1369724292