Unknown cause lockloss

However our two guesses are:

• Abrupt start/stop of a bee forklift
• Glitch at EY (which saw the lockloss first

Reminder that for the past month HAM1 pressure was being reported by a temporary "H1" version of the old cold-cathode PT100A (called H1:VAC-LY_X0_PT100B_PRESS_TORR) which uses an ADC in h0vacly and calcuates the pressure from the raw voltage signal.

Yesterday Patrick installed the h0vaclx full Beckhoff readout of this gauge via an ethernet connection. The channel name for this gauge from now onwards is H0:VAC-LX_X0_PT100_MOD2_PRESS_TORR

For now I'm showing both channels on the Vacuum MEDM and FOMs, H1:VAC-LY_X0_PT100B_PRESS_TORR upper, H0:VAC-LX_X0_PT100_MOD2_PRESS_TORR lower.

Note the H1 channel is a bigger number, the voltage signal increased slightly when Gerardo plugged the ethernet cable into h0vaclx yesterday.

TITLE: 06/18 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 3mph Gusts, 1mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 11:57 UTC

Looks like we were able to recover from a 5.8 Mexico quake fully automatically last night.

TITLE: 06/18 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 145Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

Currently Observing at 145 Mpc and have been Locked for almost 2 hours.Everything looking good.

Two hours into our previous lock, our range was slowly dropping and we could see that SQZ wasn't looking very good, and trending back the optic allignments Camilla saw that ZM4 and ZM6 weren't where they were supposed to be because the ASC hadn't been offloaded before all the updates earlier today(ndscope1). To fix this, we popped out of Observing and turned on the SQZ ASC for three minutes, which made the sqzing better(ndscope2)! This better sqzing has persisted to this next lock stretch.

After the 01:30 lockloss (85145), I sat in DOWN for a couple minutes while restarting a few Guardian nodes since awg was restarted earlier today and this should help with the guardian node error (like in 85130) (these are the ones I could think of that probably all use awg): ALS_{X,Y}ARM, ESD_EXC_{I,E}TM{X,Y}, SUS_CHARGE, PEM_MAG_INJ Tagging Guardian
 

LOG:

23:30 Observing and have been Locked for over 1 hour
00:24 Popped out of Observing quickly to run the SQZ ASC for a few minutes
00:27 Back into Observing
01:30 Lockloss
    - Sat in DOWN for a couple minutes while I restarted some Guardian nodes
    - We ended up in CHECK_MICH_FRINGES but that didn't help, so I started an IA
03:05 NOMINAL_LOW_NOISE
03:08 Observing                                                                                                          

Lockloss @ 06/18 01:30 UTC

The DAQ was restarted for a second time today at 12:41 (0-leg) and 12:50 (1-leg) for three reasons:

1) h1psldbb had a bad DAQ status following model change at 09:15 to remove an IPC sender

2) H0EPICS_VACLX.ini has new PT100 HAM1 gauge channels

3) H1EPICS_DIGVIDEO.ini has new CAM37 ISCT1-REFL channels (camera was added recently)

I regenerated H1EPICS_DIGVIDEO.ini using generate_camera_daq_ini.py, which correctly added CAM37 channels as a new server machine, but reverted CAMs[21, 23, 24, 26] back to "old" server settings. Remember these cameras were moved to a new server, but lack of masking forced them back to the old server (digivideo2). At this point the EDC was expecting a new set of channels, so as a work around I built a dummy ioc to simulate these channels to keep EDC happy. So to make a long story short, these cameras are back with their original PV set and digivideo_dummy_ioc is no longer needed. To test this I shut down the IOC with no loss of EDC connections, and verified tha the Guardian CDS_COPY "from" list was still functioning.

To run the test that calculates how much loss we have between the ASC-AS_C (anti-symmetric port) and the DCPDs at the output of the OMC (where strain is derived from).

1. This test is set up to run in NLN.
2. Turn off OMC ASC. Go to sitemap -> OMC -> OMC Control and note down the MASTER GAIN value in circled section in attached photo, then change this value to 0.
3. Go to /ligo/gitcommon/labutils/darm_offset_step and run:

conda activate labutils

python auto_darm_offset_step.py

The script turns on one notch in the DARM LSC loop, then changes the PCAL line heights so all the power is in just two frequencies.

At the end it reverses both these changes.

It can be stopped using Crtl-C but this will not restore the PCAL line heights back to their default values so you may have to use time machine.

After setting up the PCAL the script steps the DARM offset level in 120 s steps, I think it takes about 21 minutes to run.

After the script finishes, please put the OMC ASC MASTER GAIN back to its original value.

TITLE: 06/17 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 142Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 4mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

We are Observing and have been Locked for almost 1.5 hours. SQZ doesn't look good, so I might pop out of Observing in another hour to run sqz scan align. Also, if we lose lock I will be restarting a couple guardian nodes that use AWG since that was restarted earlier today, and we've already had one awg error pop up in ALS_YARM (85130).

TITLE: 06/17 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 140Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 22:26 UTC

TUES Maintenance Activities:

We had a very productive maintenance day during which the following was completed

• LVEA Laser SAFE Transition - OPS
• RCG Upgrade and misc CDS work - CDS
  • WP 12624, WP 12625, WP 12626, WP 12627
  • Alog 85120, 
• VAC Channel Work - CDS 
  • Alog 85134
• Anchor drill work - VAC
• Laser Curtains around HAM1 replaced
  • Alog 85125
• Gauge replacement prep - VAC 
• ISCT7 Poke Homodyne - ISC
  • Alog 85116
• HAM ISI Measurements - SEI - 
• FAC Work (Bees, Tech Cleaning, Inventory, Lifts, Monitors) 
• 3IFO Visit Support - 3IFO
  • WP 12628

LVEA was swept - alog 85128

*Only listing completed WP’s - close your permits if work is done.

Lock Reacquisition:

Lock recovery was quite rough due to a few issues

1. ITMY Oscillation - Whilst locking ALSY, we noticed that it was oscillating at ~3Hz. No alarms had sounded other than the CDS/Maint work’s ISI/SUS trips expected earlier in the morning that were since successfully reverted. Ryan C discovered that the L1 BOSEMs were also oscillating independently of ALS WFs (which was the initial hypothesis). Then we found a worrying increase in the IY reaction chain gain (which showed an apparent increase of a few ten thousand!). This appears to have started when the CDS upgrade was done even though the right values were save into the SDF SAFE files. CDS advised that any signals from this time are not to be trusted, but clearly something must have moved since the 3Hz signal we were still getting was still ringing (albeit ringing down for the last 3 hrs). We managed to find the appropriate filter banks, IY_R0_L2DAMP_R_GAIN and IY_R0_L2DAMP_P_GAIN and identified them as the culprits, turning them off. After the involvement of Ryan C, Tony, Sheila, Elenna, Keita, Jonathan, Dave, EJ, Oli and myself, we came to the conclusion that we were going to test turning on the gain (nominally set to 1.5 and -2.5 not 10k) with a ramp. We did this and there were no issues. Weird. So essentially CDS is saying that there was no gain and power cycling the gain fixed the IY motion which was seen IY L2, IY L1 OSEMs as well as ALSY during ALS. Elenna still thinks something is weird with IY Reaction Chain so is investigating. Picture of Tony’s IY Power Spectrum Below. (alog 85135)
2. ALS Y Fault during ScanAlign. ALSY Faulted during scan-align causing a guardian node error. I had to manual out to continue locking. This is now a known issue whereby ALS scan align (not specific to X or Y) fails due to an AWG issue. The SUSCHARGE and ALS guardians will be restarted by Oli once we have our next Lockloss. (alog 85130)
3. IMC Locking Issues. The CDS work today threw the IMC out of alignment so we had to move MC1, MC2, MC3 and PZT sliders to a time where we were successfully auto-locking.
4. SEI states, PSL and SUS were all transitioned to their SAFE (Safe State Transition Wiki) and then transition out (Safe State De-transition Wiki)
5. SDF Screenshots Linked

LOG:

Trend of H1:IOP-SUS_ITMY_WD_OSEM1_RMSOUT shows increased motion during the 10 minutes post-RCG upgrade that OMC0, see alog 85120, was clobbering IPCs, including two peaks.

The attached screenshot has cursors at the approximate start and end of OMC0 clobbering IPCs.  RMS remained high until guardian was started 30 minutes later, after which ITMY continued to ring until guardian was again restarted.

We will attempt to trace the clobbered IPCs to see if they plausibly could have driven ITMY.

WPs: 12577 and 12608
Previous work: alog 84871

This portion of the first work permit has been completed: "Migrate h0vaclx from the svn repo to the git repo. Also migrate it from using the C# code to the PowerShell code for generating the TwinCAT 3 Visual Studio solution. This would make it match h0vacmr and h0vacly in both of these regards. Update the PLC code on h0vaclx to add PT100 as an Inficon BCG552 gauge."

The installation of TwinCAT 3 on h0vaclx has been updated to version 3.1.4024.35. The PowerShell script to generate the TwinCAT 3 solution has been changed to use the TwinCAT XAE Shell instead of Visual Studio 2010 because I could not get it working with the latter. The new Inficon BCG552 EtherCAT gauge on HAM1 is connected and being read into EPICS. The code being used for the scripts is at commit d27c3ebfb424572f3aba003744e97e947e5a4873 in the git repo at https://git.ligo.org/cds/ifo/beckhoff/lho-vacuum. The shortcut in the TwinCAT autostart folder to start the EPICS IOC has been updated to point to the location of the checkout of this repo. The shortcut on the Desktop has similarly been updated.

Timeline of work:

9:56 Stopped the EPICS IOC. Set the TwinCAT runtime to Config. Started the installer for TwinCAT 3.1.4024.67.
10:02 A Windows Security dialog message appeared three times: "Windows can't verify the publisher of this driver software". Clicked "Install this driver software anyway" each time.
The installer took a very long time on "Installing Microsoft .NET Framework 5.6.1 Full".
10:22 The computer spontaneously logged me out during the installation.
10:24 Logged back in.
10:26 I started the installation of TwinCAT 3.1.4024.67 again and then soon canceled it.
10:35 I started the installer for TwinCAT 3.1.4024.35.
10:50 I restarted the computer to complete the install as prompted.
10:52 Logged back in. The installation appeared to be successful. I tried to generate the TwinCAT 3 solution from the scripts. I could not find a way around an error saying that the project template could not be found, despite it being at the path shown.
11:19 I ran 'shutdown /r' to restart the computer and try again but still got the same error about the template after the restart. I changed the script to use the TwinCAT XAE Shell instead of Visual Studio 2010. The script froze. I logged out and back in. The script succeeded in generating the solution. I scanned for terminals and did not see the BCG552 gauge on HAM1. Gerardo told me it was not connected and went to connect it. The gauge showed up and everything appears to be working. I updated the paths in the scripts for the IOC and the shortcuts to start the IOC. I checked in all of the changes to git.

We keep having some SDF observe diff for the roll mode damping gain ramp time. I added a line to the DAMP BOUNCE ROLL guardian state to set the gain to 10 seconds when the gain is set. Apparently we've been engaging the damping with no gain ramp. Hopefully this observe diff will stop popping up now.

Operator Was trying to lock in LOCKING_ARMS_GREEN[12] and ALS_YARM Faulted when It was trying to do SCAN_ALIGNMENT[57].

We Manual'd ALS_YARM back to UNLOCKED[-28]  then reloaded ALS_YARM.
Operator then took ISC_LOCK to GREEN_ARM_MANUAL[13]


Error Log:

 ALS_YARM W: RELOADING @ SCAN_ALIGNMENT.run
2025-06-17_21:15:15.740544Z ALS_YARM [SCAN_ALIGNMENT.run] Generating waveforms
2025-06-17_21:15:16.229352Z ALS_YARM [SCAN_ALIGNMENT.run] Start time is 1434230144
2025-06-17_21:15:16.229580Z ALS_YARM [SCAN_ALIGNMENT.run] Opening awg streams
2025-06-17_21:15:16.232982Z awgSetChannel: failed awgnewchannel_1(chntype = 1, arg1 = 0, arg2 = 0, awg_clnt[99][0] = 31370384) H1:SUS-TMSY_M1_TEST_P_EXC
2025-06-17_21:15:16.232982Z Error code from awgSetChannel: -5
2025-06-17_21:15:16.233280Z ALS_YARM W: Traceback (most recent call last):
2025-06-17_21:15:16.233280Z   File "/usr/lib/python3/dist-packages/guardian/worker.py", line 494, in run
2025-06-17_21:15:16.233280Z     retval = statefunc()
2025-06-17_21:15:16.233280Z   File "/opt/rtcds/userapps/release/als/common/guardian/ALS_ARM.py", line 1525, in run
2025-06-17_21:15:16.233280Z     ex.open()
2025-06-17_21:15:16.233280Z   File "/usr/lib/python3/dist-packages/awg.py", line 595, in open
2025-06-17_21:15:16.233280Z     raise AWGStreamError("can't open stream to " + self.chan \
2025-06-17_21:15:16.233280Z awg.AWGStreamError: can't open stream to H1:SUS-TMSY_M1_TEST_P_EXC: Error setting up an awg slot for the channel (-120)
2025-06-17_21:15:16.256804Z ALS_YARM ERROR in state SCAN_ALIGNMENT: see log for more info (LOAD to reset)
2025-06-17_21:15:35.011679Z ALS_YARM MODE: MANUAL
2025-06-17_21:15:35.017804Z ALS_YARM MANAGER: removed
2025-06-17_21:15:41.635373Z ALS_YARM MODE: AUTO

Late entry from last week. Roger's Machinery completed the new Kobelco installation and startup last Tuesday.  Our last task before acceptance testing of the system was to replace the filter elements in the filter tree.  When we opened the housings, we discovered that one of the elements was mangled and had evidence of rough tool use on it.  When we removed the element, we could see that there was some evidence of galling in the threads where the element screws into the filter housing (see pics).  We will order a new housing and replace it before continuing with testing the system.  

We have completed the upgrade of H1 frontends to RCG5.5.0 at 09:51.

Detailed alog will be written but surprises found/re-remembered:

EX dolphin frontends need to enable an unused port on the EY switch because the EX switch has no port control (damaged in Apr power outage)

PSL DBB model had an obsolete Dolphin IPC sender back when it used to have a dolphin card. New RCG doesn't allow senders with no cards. Removed the sender from the model, a DAQ restart is pending for this model

We had upgraded h1omc0 to 5.5.0 some time ago, but the H1.ipc file had changed so it needed a restart. Prior to restart it was clobbering the SWWD IPCs between sush34 and seih23, seih45.