h1guardian1's /usr/share/zoneinfo/leap-seconds.list file expired 28 Dec 2024 resulting in guardlog errors.

I updated the tzdata package on h1guardian1 which resolved the problem. I verified the CDS workstations already have updated leap-second files (they expire 28 Jun 2025).

TITLE: 12/28 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 34mph Gusts, 24mph 3min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.56 μm/s
QUICK SUMMARY:

Still some logic issues with H1 after the HAM6 Vacuum Gauge Failure / Interlock change where ISC_LOCK is stuck at CHECK AS SHUTTERS.  I "manually" ran a test of the Fast Shutter via the AS Port Protection medm and pushing the TEST button (it completed and was OK, also get a "HAM6 Verbal alarm which I assume is the Fast Shutter being cycled open/closed and seen by the ISI).  I went to the FAST SHUTTER Guardian, but it was MANAGED by ISC_LOCK, so I took it to AUTO and ran an INIT to get it to READY (green state), but now this node is "stolen" by USER (me).......Aaaaand, we are still stuck at CHECK AS SHUTTERS.......Aaaand....

We continue to fly blind because Guardian Logs have not been available since yesterday due to "leap second" issue.

BUT---we are still stuck at CHECK AS SHUTTERS, so I did a "MANUAL" on ISC_LOCK from CHECK AS SHUTTERS and went to PREP TR CARM; then returned to AUTO for ISC LOCK......

....Then there was a  LOCKLOSS at CARM TO ANALOG at 1538. 

Lockloss is not surprising since we are still at high microseism (above 95th prcentile) AND for the last 6hrs winds have been between 20-40+mph.

13:00 UTC lockloss, the lockloss tool is not updating. Appears to be from an ITMX CPS glitch, the suspension tripped.

We're at CHECK_AS_SHUTTER with a notification about checking if the shutter worked on the last lockloss, but I can't open any GRD logs or any MEDM all states screens. I'm getting :

raise RuntimeError(f"Error loading leap file: {path}")
RuntimeError: Error loading leap file: https://www.ietf.org/timezones/data/leap-seconds.list

TITLE: 12/28 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently Observing and have been Locked for 4 hours. Relocking went well, although we were in OMC_WHITENING for a bit damping violins. I did not have to touch anything besides choosing HIGH_ARM_POWER on the LOCKLOSS_SHUTTER_CHECK guardian (I think that was a side effect of me MANUALing it before correctly INITing it into LOW_ARM_POWER earlier in the locking process - I had reassigned the node manager back to ISC_LOCK but I think it must've already passed the part where ISC_LOCK told it to go to high power) and the evening has been quiet.
LOG:

23:16 Lockloss while sitting in LOWNOISE_ASC, going to sit in IDLE

00:19 Started relocking

02:00 NLN

02:01 Observing                                                                                                                                                                                                                               

TITLE: 12/27 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 8mph Gusts, 5mph 3min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.78 μm/s
QUICK SUMMARY:

Currently starting relocking and at ENGAGE_ASC_FOR_FULL_IFOjust lost lock from ENGAGE_ASC_FOR_FULL_IFO. Fil just finished the work reconnecting the ITM ESDs and connected the HAM6 high voltage relays to the BSC3 interlocks(something like that - see attached). After he did this we tested out the port protection but it failed due to the fast shutter not being open. It seems like the fast shutter usually reopens when we are going through PRMI/MICH states, so that might have been why it was still closed from our last lockloss, but it doesn't completely make sense because earlier before Corey ran the test, the shutter had opened on its own while we sat in IDLE. I opened the shutter medm and manually opened the shutter and reran the test. It failed again and said it was due to the fast shutter not being able to reopen even though it had correctly closed and reopened it. I toggled the shutter closed and back open, and this time the test ran and came back with OK, so it seems like we are good to go to high power. Jenne called and had me check that we had ESD outputs on ITMX, which we did so it looks like everything is good to go.  When at CHECK_AS_SHUTTERS we did have the LOCKLOSS_SHUTTTER_CHECK alert that we needed to manually check the shutter, so I ran the test script on the AS port protection screen again just to make sure and then INIT'ed that guardian.

As as of Fil completing this work, THE ENTIRE CORNER STATION HIGH VOLTAGE INTERLOCK SYSTEM(except SQZ) IS CONNECTED TO BSC3. tagging VAC, SYS

Also, I am unable to view the guardian logs - when I open them they say 'Leap second data is expired' and then close Tagging CDS

TITLE: 12/27 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Today started off with addressing the HAM6 interlock which was no longer functioning due to missing the vacuum gauge it needs (i.e. the failed PT110 on HAM6 from last night!). 

Fil came in to take the BSC3 vacuum gauge to take the FAILED HAM6 vacuum gauge's place for the HAM6 interlock.  This would pull the BSC3 gauge from the interlock it was used for:  ITM ESDs---it was thought the ITM ESDs were NOT used.   Returned to locking, but then it was later brought up that the ITM ESD interlock is indeed needed for TRANSITION FROM ETMX.  

So we were on hold again.  

Ultimately it was decided to use the BSC3 vacuum gauge for BOTH the (1) HAM6 & (2) BSC3's ITMx ESD interlocks.  Fil came in (2nd time driving out to the site today!) to make this change in the afternoon.  

The hope is we are now back to a the state before the HAM6 gauge/interlock crashed last night, and Oli can return with locking.  Winds are lower than yesterday, BUT microseism is still high.  

LOG:

• 1700-1752 Fil working on a bypass for the HAM6 Interlock (for Fast Shutter + PZT)
  • 1756 Explicitly ran the AS Port Protection Test (sitemap/SYS/AS Port Protection) and it received an OK
• 1804 Marked Observatory Mode to CORRECTIVE MAINTENANCE, but it really should have been this at 959pmPT
• 1807-1904 Initial Alignment + Manual Alignment
  • Received ISC LOCK node in error.
  • LOAD cleared this up, and IA started (green arms were off so a Scan Alignment was run for both arms)
    • Had TMS x & y saturation verbal alarms for green arm alignment (new to me)
    • Needed to touch up SR2 /SRM
      • Actually SR2 was not touched since AS_C looked centered, but SRM was touched up a bit (mostly negative Pitch)
  • Ran a Manual Alignment of the SRC a few times
• 1904 Locking---made it to LOWNOISE_ESD_ETMY (Tried this new state because of the ITM ESD change for the HAM6 Interlock), but...
  • 2052 Lockloss when we tried this LOWNOISE_ESD_ETMY
  • We hung out at LOWNOISE ASC while Sheila and then Jenne started looking at ESD stuff, because it turns out that we DO use the ITM ESDs
• 2127-2143 Manual Initial Alignment (basically skipping arms and focusing on DRMI) because arms looked fine for this lock, but DRMI & PRMI looked badly misaligned.
  • 2135 Jenne mentioned there's a possibility Fil may come out again to get the ITM ESD interlock back online.
  • Fil arrived while I was doing the Site Walkthrough and he touched base with Oli.
• 2224-2315 Site Walkthrough (corey)
• 0000-0027 Dripta in PCal Lab.

[Corey, Sheila, Dave, Jenne]

With the ITM ESD high voltage interlock being used instead for HAM6 high voltage things, we cannot use the ITMs for the ESD transition at the end of the locking sequence.  We are trying to instead use ETMY. 

• Sheila had started work on modifying our usual guardian states, to allow us to use the L3 on ETMY.  That was in progress, when we decided to try the prewritten (but not tested in while) 'lownoise esd etmy' states.
• We tried using the 'lownoise esd etmy' state, but lost lock when it was doing the L3 Lock L 5 second ramp over to using ETMY.
  • Yesterday's lockloss (at least the one around 27 Dec 2024 02:03:00 UTC) was, as Corey said, successful getting off of ETMX, and then failed at swapping back to the low voltage version of ETMX.
  • Today's lockloss had a ringup of about 161 Hz.
• Richard called to check in, and said that it shouldn't be too hard to tee off (or something) that interlock cable, so that it can be used for both ITM ESDs and also the HAM6 high voltages.  Richard is planning to talk to Fil about the feasibility of this.
• Since our current plan is to see if we can go back to our nominal locking sequence, I have copied over Sheila's temporary guardian changes (attached here, also), and done an SVN revert to the version of ISC_LOCK.py that RyanS had checked in just before the winter holidays.  The only diffs between that version and the version in the userapps directory were related to today's temporary changes.  Corey has reloaded the guardian.
• Richard checked with Fil, and Fil is on the way to make the change, so we can go back to using our usual locking sequence.

Fri Dec 27 10:16:08 2024 INFO: Fill completed in 16min 4secs

(Dave Barker, Fil Clara, Jenne Driggers, Corey Gray, Gerardo Moreno, plus those who signed work permit and whom I phoned last night)

Last Night Recap:
Last night at 959pmPT, there was an audible alarm for the HAM6 vacuum gauge (PT110).  Gerardo went out to check to see if it was dead.  He phoned Fil and they did a power cycle of the gauge to see if this would improve the situation, but it did not.  After this, H1 was taken to IDLE for the night (OWL shift cancelled).  (Should have taken Observatory Mode to CORRECTIVE MAINTENANCE at this point, but only remembered to do this this morning.)

Work Permit 12260 generated.  Fil arrived on-site around 9amPT (17utc).

This Morning's Summary of Fil's Work:

The HAM6 Interlock (for operations of the Fast Shutter + PZT) is now Using the BSC3 Interlock (or VAC gauge from BSC3 since it was decided to BYPASS the HAM6 VAC gauge which failed last night [alog 82005]).  This means that the BSC3 Interlock is no longer being used for the ITM ESDs (which sounds like it's fine because Jenne/Fil said we don't use the ESDs for the ITMs).

Fil noted that he confirmed that the Fast Shutter Chassis in the LVEA is OPERATIONAL.  Fil then left the site after giving me this summary.

As far as a test of this new HAM6 Interlock configuration, I intentionally ran a test of the AS Port Protection system (sitemap/SYS/ AS Port Protection....this was my first time doing this).  After clicking the RUN (test) button, after a few seconds received an "OK" for the (1) Test & (2) Fault.  After this test and the signed-off Work Permit, we are OK to GO for High Power locking.

Currently running an Initial Alignment (note:  ISC_LOCK initially went into ERROR [red box] when I selected Initial Alignment (screenshot of error message attached).  LOAD-ing ISC_LOCK (suggested in the Log) cleared this up.

ALSO: 

• Attaching a screenshot of the AS Port Protection medm
• CDS Alarm box on the CDS Overview remains (related to the lingering issue of the failed PT110 gauge).

TITLE: 12/27 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 10mph Gusts, 8mph 3min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.69 μm/s
QUICK SUMMARY:

Filibert has been on-site this morning to work on bypassing the failed vacuum gauge at HAM6 to get the Fast Shutter interlock operational again (just received a summary of his work).

Next step is to wait for the Work Permit to get signed for high power operations and then return to locking.

Microseism is still high but only light winds (but basically this is the same state we were in last night where H1 had troubles getting past TRANSITION FROM ETMX.  SEI CONFIG is in the USEISM state.

Since the HAM6 gauge which failed at 10pm local time is used as part of the interlock system for the Fast Shutter, Jenne made the call to have the H1 down for the night (Owl Shift Canceled)

• H1 in IDLE with the IMC Locked.

Tomorrow we will need to figure out the plan forward for H1 operations.  Whatever the fix will require a Work Permit and will need to figure out availability for who can do this work.  

TITLE: 12/27 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Wind
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

Bit of a frustrating shift with H1 losing lock most of the time at the same point late in the ISC_LOCKing scheme (alog 82000).  Plus dealing with high seismic motion (microseism is still around 1um/sec).  No alignments were run or needed this shift.  With locking, made it fairly repeatedly all the way up to around TRANSITION_FROM_ETMX / LOWNOISE_ESD_ETMX for majority of shift's locklosses.  For relocking, DRMI generally looks horrible, but PRMI does the trick for continuing on beyond DRMI---hence no Initial Alignments have been run this evening.

And shift ends with a M5.9 mid-Atlantic earthquake rolling through.
LOG:

• 0048-0115+ Site Walk thru (tony)
  • Tony noted that in the H2 Room there was a Test Stand computer which was beeping & had a red light flashing (hadn't heard this on previous walk-thrus).
• 0055 LOCKLOSS at LOWNOISE_ESD
• 0314 Observing
  • 0325 LOCKLOSS
• 0424 LOCKLOSS @TRANSITION_FROM_ETMX
• 0534 LOCKLOSS @LOWNOISE_ESD_ETMX
• 0546 Incoming Earthquake (M5.9 mid altlantic )
• Various ALS & DRMI locklosses to end the shift.
• 0559 VACUUM Alarm Handler alarm for LVEA's PT110.
  • Looking at CDS Overview:  red VAC & CDS Alarm blocks
  • Dave contacted me via Teamspeak to let me know about the above HAM6 alarm PLUS an alarm for BSC3 (which is not as clear and Dave is still investigating)
  • I contacted Janos to apprise him of these alarms.  (he thought we lost the gauge for HAM6 and said that BSC3 might be a glitch (since no other gauges were alarming).  At any rate, he was going to contact Dave.
  • Coincidentally, Gerardo just happened to arrive within 20min of these alarms (he was in the area because he needed to pick up a Yubikey!  So now he can take a look at these gauges and chat with Dave!
  • Also been in contact with TJ, Jenne, and Fernando.

At 21:58 VACSTAT issued a BSC3 alert, which looks like it could be a sensor spike. 1 minute later PT110 (HAM6) sensor went offline to flat 0.0Torr.