david.barker@LIGO.ORG - posted 10:26, Monday 15 January 2024 (75403)
Mon CP1 Fill
Mon Jan 15 10:03:32 2024 INFO: Fill completed in 3min 30secs
Images attached to this report
Mon Jan 15 10:03:32 2024 INFO: Fill completed in 3min 30secs
TITLE: 01/15 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 6mph Gusts, 5mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.34 μm/s
QUICK SUMMARY:
When I walked in H1 was in MAX_POWER Violins look reasonable.
FMCS Temps are no longer working again. Looks like Dave restarted it again last night and it only worked for an hour or 2 then failed again.
IOC froze at 23:39 PST Sun, I restarted it 00:35 Mon. It had ran 13h 34m since last restart.
TITLE: 01/15 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Currently Locked for 15 minutes and Observing. Violins have been great my entire shift, although I did end up needing to turn down the damping for ITMY05 due to ITMY20 starting to go back up after 6 hours of going down(attachment1). The other three modes were still decreasing.
Our range has been extremely noisy(attachment2) so Jenne suggested I run the rubbing script, but it looks like nothing is rubbing(attachment3).
One lockloss during my shift(75397) and a bit of trouble relocking - I needed to move ITMX due to ALSX not being able to get above 0.6 and refusing to lock on TEM00 despite ~30minutes of adjusting ETMX/TMSX P/Y. I followed the troubleshooting guide(G1602280) for Initial Alignment Green Arms and was able to confirm that the WFS error was related to the positioning of the beam on ITMX(attachment4 - orange trace on top left), and so I adjusted ITMX yaw.
Later on during LASER_NOISE_SUPPRESSION, the detector got stuck due to the ISS diffreacted power being too low, but Ryan S was able to advise me on how to raise the diffracted power and now here we are Observing!
LOG:
00:00UTC Detector Locked for 13 hours and Observing
00:28 Taken out of Observing due to SQZ unlock
00:31 SQZ relocked itself and we went back into Observing
04:13 Lockloss (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=75397)
04:15 Initial Alignment
- Took a while due to ALSX not being able to correctly catch; I followed the troubleshooting guide(https://dcc.ligo.org/LIGO-G1602280) for Inital Alignment Green Arms to adjust ITMX
07:09 Stuck in LASER_NOISE_SUPPRESSION due to low diffracted power in the ISS, but we continued up once I bumped it up
07:42 NOMINAL_LOW_NOISE
07:42 Observing
The annulus ion pump on BSC8 railed at about 06:22 utc today, the main vacuum pressure was checked and no change noted.
Pump will be replaced as soon as possible, for now no action is required.
Working on relocking. Can't get ALSX higher than 0.6 and won't catch on the 00 mode (and ndscoping H1:ALS-X_CAM_ITM_{PIT,YAW}_ERR_OUT16, H1:ALS-X_WFS_A_I_{PIT,YAW}_OUT16 , and H1:ALS-X_WFS_B_I_{PIT,YAW}_OUT16 is showing the ITM is needing to be moved to get the dof3 error within range, so working on that.
TITLE: 01/15 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY:
Detector has been Locked for almost 13 hours and we are Observing. Winds below 10mph and useism is relatively low. Smaller temperature fluctuations in the LVEA and we are coming back down from a large temperature excursion at EX.
Range is super glitchy and omicron has been flagging lots of lines of louder glitches between 100 and 700Hz.
TITLE: 01/15 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
H1 Currently Locked and Observing.
The EX Temps have fallen back down to a reasonable temp and are looking better.
I haven't had a problem with the Temp data since Dave Restarted the IOC.
Range is still looking glitchy and weird.
LOG:
No Log
Laser Status:
NPRO output power is 1.835W (nominal ~2W)
AMP1 output power is 68.58W (nominal ~70W)
AMP2 output power is 139.7W (nominal 135-140W)
NPRO watchdog is GREEN
AMP1 watchdog is GREEN
AMP2 watchdog is GREEN
PMC:
It has been locked 2 days, 5 hr 16 minutes
Reflected power = 17.82W
Transmitted power = 109.6W
PowerSum = 127.4W
FSS:
It has been locked for 0 days 11 hr and 40 min
TPD[V] = 0.5445V
ISS:
The diffracted power is around 1.8%
Last saturation event was 0 days 11 hours and 41 minutes ago
Possible Issues:
FSS TPD is low
ISS diffracted power is low
STATE of H1: Observing at 153Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 6mph Gusts, 2mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY:
While trying to find a way to damp Violins with Non-Nominal Gains without Violin Guardian undoing my changes. I accidentally took us out of OBSERVING for a few seconds by taking the Violin manager to Damp on Simple.
18:46 UTC Dropped from OBSERVING
18:47 UTC Back to OBSERVING
But Since then I've been able to damp ETMY Mode 1 and Mode 20 with the Settings from Rahul's alog last night.
Original post: SEI log 2374.
I updated the filter used for the picket fence, it strongly rejects the microseism now, hopefully this will make the alerts from the picket fence more reliable. The filter adds 5 seconds of delay to the waveforms, but given the short latency of the NEIC and PNSN seedlink connections, we should be able to still get good performance.
Also, I noticed that UO_LAIR was not giving any signal for the LHO picket fence, so I swapped it for UO_DING (located in Powers, OR), which is a very similar station.
To get the update, please git pull at the picket fence folder
Sun Jan 14 10:10:46 2024 INFO: Fill completed in 10min 42secs
The FMCS EPICS IOC froze up late last night about two hours after I restarted it. Timeline is:
IOC console as found is attached.
Opened FRS30184
TITLE: 01/14 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 3mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.28 μm/s
QUICK SUMMARY:
When I walked in H1 was in OBSERVING for 4 hours, but we had reasonably large 500hz Violin spike on darm, and the DCPD's were diverging.
I quickly checked the Violins, and noticed that the ITMY Mode 5 seting was in it's Nominal Gain state of +0.02 and was rising again.
Changed the Gain back to -0.02 and it has already turned around and started to fall.
Im not sure what is going on with temperature senors looks like there was a rest or something but the temps all seem to be "Frozen" or not changing like they normally do.
I dont see any CDS errors about this, and it doesn't seem to be mentioned in the alogs. So tagging FMCS and documenting the temps are they currently stand.
CS: Everything looks good in the LVEA all green.
H0:FMC-CS_LVEA_ZONE1B_DEGF= 67.4
H0:FMC-CS_LVEA_ZONE2A_DEGF= 66.79
H0:FMC-CS_LVEA_ZONE3A_DEGF = 67.3
H0:FMC-CS_LVEA_ZONE4_DEGF =67.7
H0:FMC-CS_LVEA_ZONE5_DEGF =67.3
EX Some temp channels are RED:
H0:FMC-EX_VEA_202A_DEGF = 57.40 RED
H0:FMC-EX_VEA_202B_DEGF =68.5 green
H0:FMC-EX_VEA_202C_DEGF =68.3 green
H0:FMC-EX_VEA_202D_DEGF = 58.70 RED
H0:FMC-EX_VEA_AVTEMP_DEGF = 64.73 green
EY:
H0:FMC-EY_VEA_202A_DEGF = 61.7
H0:FMC-EY_VEA_202B_DEGF = 67.40
H0:FMC-EY_VEA_202C_DEGF = 67.6
H0:FMC-EY_VEA_202D_DEGF = 59.9 RED
H0:FMC-EY_VEA_AVTEMP_DEGF = 64.97
Since this morning, I've tried, restarting the ndscope, and the nuc.
And slowly figured out that its not an FMCS error.
Also I stand corrected I didn't see Dave's alog abou this exact issue last night .
Dave has restarted the IOC in hopes of recovering the temperature information for us.
If that doesn't continue to work, then I will make an ndscope of the Dust Mon Temps and put that up on the FOM for screenshots instead.
Thanks Dave!
H1 called for assistance at 11:28 UTC since the filter cavity was having trouble locking again. Much like what Oli needed to do four hours ago (alog75384), I paused the SQZ_FC Guardian, locked the FC green VCO manually using the toggle switch on the common mode board, and optimized SQZ-FC_TRANS_C_LF_OUTPUT using FC2 P/Y sliders (and successfully bringing the signal to around 83 counts). After resuming the Guardian and re-requesting SQZ_MANAGER to FREQ_DEP_SQZ, the FC locked without issue, I accepted the new slider values in SDF (screenshot attached), and H1 was able to start observing at 12:03 UTC.
Contrary to the issue earlier tonight, I did not see the CLF DEMOD phase go to zero (it's currently hanging around 171), so I assume the ADF angle servo is working properly this time.
TITLE: 01/14 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Struggling with violins and relocking all day, but we were able to reach NOMINAL_LOW_NOISE twice, but lost lock the first time before we could go into Observing (75381). Currently in NLN and Vicky and Daniel are working on getting the SQZ angle to behave because it keeps trying to move towards 0.
Violin modes to keep an eye on: ITMY05/06, ITMY08, ETMY01, ETMY20 (Rahul's alog with some values that work: 75378)
LOG:
00:00UTC Detector Relocking and in MAX_POWER trying to damp violins
03:24 Continuing up towards NLN
03:36 Reached OMC_WHITENING
04:48 Went into NOMINAL_LOW_NOISE
05:10 Lockloss from NOMINAL_LOW_NOISE (Not Observing) (75381)
05:30 Lockloss from LOCKING_ALS
05:41 Lockloss from CHECK_MICH_FRINGES, starting an Initial Alignment
06:10 Initial Alignment completed, relocking
06:56 Reached MAX_POWER, pausing to damp violins
07:42 Reached NOMINAL_LOW_NOISE
- Had to realign FC2 (72084)
08:11 OBSERVING!
08:15 Dropped out of Observing to troubleshoot SQZ Angle issue
Some SQZ troubleshooting notes:
H1 is now back to observing at 156Mpc following the SQZ troubleshooting described above as of 08:48 UTC.
There is notes on getting FC to lock in the Troubleshooting SQZ wiki. There were lots of LVEA temperature swings this weekend, probably causing trouble in FC alignment.
I'm unsure why H1:SQZ-ADF_OMC_TRANS_PHASE would become unstable when it previously worked for 6+ locks, plot. We could again look at moving the position of the ADF line, but couldn't get the servo stable in 75250.
The first unstable incident was from SQZ_ANG_ADJUST adjusting the angle from 170 to 115deg before the FC had locked, plot. SQZ_MANAGER shouldn't request SQZ_ANG_ADJUST to ADJUST_SQZ_ANG_ADF until it's in FDS and the FC is locked at LOCKED_IR, however at 2024-01-14 07:48:01 UTC it was requested, when SQZ_MANGER was still in SQZ_READY_IFO. This looks like someone requested it so wouldn't happen by itself.
