Ryan C, Jonathan, Dave:
Starting around 18:28 Sun 15jan2025 the control room reported name resolution issues within CDS. Also the GC WIFI went offline.
The CDS alarm system froze up at 18:28, which agrees with the time the other services went offline.
Jonathan is reporting issues contacting GC DNS and managment machines, indicating this could be a GC issue.
00:16 UTC lockloss
TITLE: 06/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Mostly quiet day with one lockloss in the afternoon and an easy reacquisition. After relocking, we've been having to touch up SQZ a couple of times. H1 has now been locked for 1.5 hours.
TITLE: 06/15 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 122Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 11mph Gusts, 6mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY:
ASC ran away and gave a notification to reset it, 23:07 we dropped observing to do so. We had to move the angle back as well as it was reset.
23:10 UTC Observing
Lockloss @ 20:52 UTC after just under 15 hrs locked - link to lockloss tool
No obvious cause.
Ran an initial alignment doing PRC align by hand, then main locking was fully automatic.
Once at low noise and before observing, I noticed SQZ looked poor, and the SQZ BLRMs were worse than at the start of the last lock stretch. I cycled SQZ_MANAGER through 'SCAN_SQZANG_FDS' to try and find a better SQZ angle, but the Guardian wasn't able to find a place that made the BLRMs and BNS range both look good, so I manually searched around at different angles and eventually settled on 139 with a "cleaned" range of 153Mpc. Started observing at 22:34 UTC.
State of H1: Observing at 151Mpc
H1 has now been locked and observing for 13 hours. Earthquake rolled through a couple hours ago, but otherwise a quiet morning.
Sun Jun 15 10:12:39 2025 INFO: Fill completed in 12min 35secs
FAMIS 26426
Laser Status:
NPRO output power is 1.852W
AMP1 output power is 70.39W
AMP2 output power is 140.6W
NPRO watchdog is GREEN
AMP1 watchdog is GREEN
AMP2 watchdog is GREEN
PDWD watchdog is GREEN
PMC:
It has been locked 27 days, 1 hr 38 minutes
Reflected power = 23.08W
Transmitted power = 105.6W
PowerSum = 128.7W
FSS:
It has been locked for 0 days 11 hr and 32 min
TPD[V] = 0.8236V
ISS:
The diffracted power is around 4.0%
Last saturation event was 0 days 12 hours and 51 minutes ago
Possible Issues:
PMC reflected power is high
TITLE: 06/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 4mph 3min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY: H1 has been locked and observing for almost 9 hours and range looks good.
TITLE: 06/15 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 145Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: We stayed locked or most of the shift, ~6.75 hours. I ran a coherence measurement, there's a lot of >10Hz CHARD_P, and <10Hz CHARD_Y. We're currently relocking at DRMI.
LOG: No log
03:52 UTC lockloss
TITLE: 06/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Only one lockloss today and two purposeful drops from observing for calibration sweeps and SQZ fixing. DARM high frequency still doesn't look amazing, so there might need to be some SQZ angle adjustment. H1 has now been locked for 2.5 hours.
TITLE: 06/14 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 20mph Gusts, 9mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.07 μm/s
QUICK SUMMARY:
I took H1 out of observing for 15 minutes starting at 22:00 UTC when I noticed the range looked low and the SQZ_MANAGER Guardian reported that the SQZ ASC AS42 wasn't on.
Reminding me of the SQZ ASC issues we've been seeing early in lock stretches, I requested SQZ_MANAGER to 'RESET_SQZ_ASC_FDS' and then back to 'FREQ_DEP_SQZ', but the SQZ_LO_LR Guardian reported low OMC_RF3 power.. I then set SQZ_MANAGER to 'DOWN' while I trended ZMs to the last lock when squeezing was good. ZM6 was very far off according to OSEMs (several hundreds of µrad in both pitch and yaw), so I moved it back to where it was at the end of the last lock and requested SQZ_MANAGER again to 'FREQ_DEP_SQZ'. This time, there were no holdups and squeezing was restored, so I promptly returned H1 to observing. BNS range seems to have recovered and SQZ BLRMs look better also.
Jonathan is heading to the site to investigate.
From the control room perspective:
teamspeak continues to run on the verbal machine.
phones continue to work
alog is accessible if the IP number is used, not the name.
scripts are failing if they need to resolve names, this is preventing squeezer work and H1's range is down to the 80s.
the alarm/alert system cannot resolve twilio's address, so no alarm texts/emails can be sent.
The issue has been resolved by power cycling the sw-osb163-0 switch. This is what DNS and a few other key services hang off of.
I restarted the switch around 8:14pm local time. Ryan C. confirms that he has access to the alog. I can get to the management machines and the dns servers, both locally and via offsite routes.
Alarms restarted itself at 20:20 and I restarted alerts at 20:54. Test messages confirmed these services are working correctly.
Opened FRS34439 to cover this, specifically how it impacted on control room operations.