WP12099 h1iscey computer swap

Erik:

Recenly we discovered that h1iscey (W2245) has a bad PCIe slot-7 on its motherboard, which is the cause of the A2 Adnaco issue in its IO Chassis. We have been skipping this slot in the past weeks.

h1iscey's front end computer was swapped with one from the test stand. See Erik's alog for details.

No model changes were required, no DAQ restart.

WP12108 laptop upgrade

Erik:

End station CDS laptops were upgraded.

Alarms reconfiguration

Dave:

The FMCS reverse osmosis (RO) has become very stable. The alarm settings were changed to alarm after 1 hour (was set to 1 day).

New CDS Overview

Dave:

New overview was released, see alog for details.

I have release a new CDS Overview, image is attached.

The previous overview was a hybrid of computer generated and hand drawn content. Specifically the upper 80% (the FE and DAQ section) was computer generated and the lower 20% was a mix of external files and hand drawn.

The new overview is completely genenerated by python code (generate_cds_overview_medm.py). This is reflected in its name change from H1CDS_APLUS_O4_OVERVIEW_CUST.adl to H1CDS-CDS_OVERVIEW_MACHINE_GEN.adl.

To open the new CDS overview, you need to kill all existing MEDMs and restart the sitemap from scratch, otherwise you will see a strange overview-within-an-overview screen.

New features:

Lower area has better layout, rows and columns align.

New IFO range "7 segment LED" section, hopefully range can be more easily read from afar. Currently it has a static orange-on-black color scheme, later version will turn green when in observe.

Color standardisation: related display buttons are light-blue on navy, shell execution buttons are light-green on dark-green.

Most areas can be clicked to open related displays.

To Do:

Complete the slow-controls and FMCS areas

The LVEA temperature trends will stabilize throughout the day today. The excursions are a result of the work I did this morning to the zone reheats. 

The TTFSS investigation is still ongoing, microseism has greatly risen to above the 90th percentile, the wind is also rising but its only ~20 mph.

There was an issue today where the VM hypervisor running alog, svn, awiki, ldap, and some services lost its network.  It has been restored.

The microseism today has reached fully above the 90th percentile, recreating Roberts plot from alog74510 the seismometers seem to show the largest phase difference is between EX and the CS, and the 2nd highest is with EY which suggests its because of the motion from our coast? From windy.com there's currently a 10 meter wave low pressure system off the coast of WA that's pretty much along the axis of the Xarm. These seismometers are also dominated by a ~0.12 Hz oscillation.

Yesterday afternoon until 2 am, we had low range because the squeezing angle was not well tuned.  As Naoki has noted 78529 this can happen when the OPO PZT is at a lower voltage than we normally operate at. This probably could have been improved by running SCAN_SQZANG. 

I've edited the OPO_PZT_OK checker, so that it requires the OPO PZT to be between 70 and 110 V (it used to be 50 to 110V).  This might mean that sometimes the OPO has difficulty locking, (ie, 76642), which will cause the IFO to call for help, but that will avoid running with low range when it needs to run SCAN_SQZANG.

No new damage noted. First pic is EX, second is EY.

h1iscey frontend server was replaced because of a possible bad PCI slot that was preventing connection to one of the Adnaco PCIe expansion boards in the IO Chassis , see here: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=80035

The new frontend is running without issue, connected to all four Adnacos.

The new frontend model is Supermicro X11SRL-F, the same as the old frontend.

Tue Oct 01 08:09:24 2024 INFO: Fill completed in 9min 20secs

VACSTAT issued alarms for a BSC vacuum glitch at 02:13 this morning. Attachment shows main VACSTAT MEDM, the latch plots for BSC3, and a 24 hour trend of BSC2 and BSC3.

The glitch is a square wave, 3 second wide. Vacuum goes up from 5.3e-08 to 7.5e-08 Torr for these 3 seconds. Neighbouring BSC2 shows no glitch in the trend.

Looks like a PT132_MOD2 sensor glitch.

1411809803

The FSS glitching isn't seen in this lock that ended a 11 hour lock, but our old friend the DARM wiggle reared its head again. (Attachment 1)

1411824447

Useism came up fast, so many of the signals are moving a bit more, but DARM sees a 400ms ringup that I don't see other places. Again, the FSS didn't seem to be glitching at this time.

TITLE: 10/01 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive maintenance
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 2mph Gusts, 1mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.73 μm/s
QUICK SUMMARY: The IFO had just lost lock at low noise coil drivers as I arrived. I put it into IDLE and maintenance is already starting. Useism has grown to over 90th percentile, perhaps the cause of the lock loss, but I still need to look into this.

Workstations anhd wall displays were updated and rebooted.  This was an OS packages update.  Conda packages were not updated.

TITLE: 10/01 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 137Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: Observing at 140Mpc and have been locked for 7 hours. Range is still low unfortunately but it's been a quiet evening, but we did get another superevent!
LOG:

23:00 Observing and have been Locked for 1 hour

23:47 Superevent S240930du

Lockloss @ 09/30 04:54UTC

We went out of Observing at 04:53:56, and then lost lock four seconds later at 04:54:01.

There is a bump disturbance in the H1 strain spectrum that is degrading the noise in the vicinity of the Crab pulsar. Attached are zoomed-in ASDs from recent dates (drawn from this CW figure of merit site). That band was pretty stable in O4a, but has been unstable in O4b. I can see hints of the bump as early as the end of March 2024 (see calendar navigation to sample dates).

I have poked around in Fscan plots and tried the handy-dandy Carleton coherence mining tool, but saw no obvious culprit PEM channels. Is there a new motor running somewhere (or an old motor with a drifting frequency)? 

Attachments:

Sample ASD snippets from August 1, September 1 & 2, along with a snippet from the end of O4a on January 16.

The red curve is from one day of averaging, and the black curve is the run-average up to that date.

The vertical bars show the approximate band of the Crab pulsar during the O4 run (taking into account Doppler modulations and long-term spin down).

Edit : Neil knows how to make links, now. Tony and Camilla were instrumental in this revelation.

This is an update to my previous post (Neil’s previous post on this topic) about looking for possible explinations why similar seismic wave velocities on-site may or may not knock us out of lock.

The same channels are used in addition to:

SEI- * ARM_GND_BLRMS_30M_100M
SEI- * ARM_GND_BLRMS_100M_300M
Where * is C, D, X, or Y.

I have looked at all earthquake events in O4b, and only ones which knocked us out of lock. This is to simplify the pattern search, for now. Here are the results.

Total events : 29

Events with ISI y-motion dominant : 11 (30-100 Hz) : 8 (100-300 Hz)
Events with ISI x-motion dominant : 3 (30-100 Hz) : 0 (100-300 Hz)
Events with ISI z-motion dominant : 9 (30-100 Hz) : 19 (100-300 Hz)

Events with ISI xy-motion dominant : 1 : 0 (Both axes are similar in amplitude.)
Events with ISI yz-motion dominant : 0 : 1
Events with ISI xz-motion dominant : 0 : 1

Events with IS xyz-motion dominant : 5 : 0

Total SEI- * ARM recorded events : 8
CARM dominant events : 7 : 8
C/XARM dominant events : 1 : 0

Conclusion is that in the 30-100 Hz band, it is equally likely to have either z- or y-axis motion be dominant. In the 100-300 Hz band, the ratio is about 1:2 for z and y motion being dominant during lockloss.

Clearly, common modes are a common (^_^) cause of lockloss.

Note that velocity amplitudes should be explored more.