WP12099 h1iscey computer swap
Erik:
Recenly we discovered that h1iscey (W2245) has a bad PCIe slot-7 on its motherboard, which is the cause of the A2 Adnaco issue in its IO Chassis. We have been skipping this slot in the past weeks.
h1iscey's front end computer was swapped with one from the test stand. See Erik's alog for details.
No model changes were required, no DAQ restart.
WP12108 laptop upgrade
Erik:
End station CDS laptops were upgraded.
Alarms reconfiguration
Dave:
The FMCS reverse osmosis (RO) has become very stable. The alarm settings were changed to alarm after 1 hour (was set to 1 day).
New CDS Overview
Dave:
New overview was released, see alog for details.
I have release a new CDS Overview, image is attached.
The previous overview was a hybrid of computer generated and hand drawn content. Specifically the upper 80% (the FE and DAQ section) was computer generated and the lower 20% was a mix of external files and hand drawn.
The new overview is completely genenerated by python code (generate_cds_overview_medm.py). This is reflected in its name change from H1CDS_APLUS_O4_OVERVIEW_CUST.adl to H1CDS-CDS_OVERVIEW_MACHINE_GEN.adl.
To open the new CDS overview, you need to kill all existing MEDMs and restart the sitemap from scratch, otherwise you will see a strange overview-within-an-overview screen.
New features:
Lower area has better layout, rows and columns align.
New IFO range "7 segment LED" section, hopefully range can be more easily read from afar. Currently it has a static orange-on-black color scheme, later version will turn green when in observe.
Color standardisation: related display buttons are light-blue on navy, shell execution buttons are light-green on dark-green.
Most areas can be clicked to open related displays.
To Do:
Complete the slow-controls and FMCS areas
The LVEA temperature trends will stabilize throughout the day today. The excursions are a result of the work I did this morning to the zone reheats.
The TTFSS investigation is still ongoing, microseism has greatly risen to above the 90th percentile, the wind is also rising but its only ~20 mph.
There was an issue today where the VM hypervisor running alog, svn, awiki, ldap, and some services lost its network. It has been restored.
The microseism today has reached fully above the 90th percentile, recreating Roberts plot from alog74510 the seismometers seem to show the largest phase difference is between EX and the CS, and the 2nd highest is with EY which suggests its because of the motion from our coast? From windy.com there's currently a 10 meter wave low pressure system off the coast of WA that's pretty much along the axis of the Xarm. These seismometers are also dominated by a ~0.12 Hz oscillation.
Yesterday afternoon until 2 am, we had low range because the squeezing angle was not well tuned. As Naoki has noted 78529 this can happen when the OPO PZT is at a lower voltage than we normally operate at. This probably could have been improved by running SCAN_SQZANG.
I've edited the OPO_PZT_OK checker, so that it requires the OPO PZT to be between 70 and 110 V (it used to be 50 to 110V). This might mean that sometimes the OPO has difficulty locking, (ie, 76642), which will cause the IFO to call for help, but that will avoid running with low range when it needs to run SCAN_SQZANG.
Reverted OPO checker back to 50-110V as we moved the OPO crystal spot.
No new damage noted. First pic is EX, second is EY.
h1iscey frontend server was replaced because of a possible bad PCI slot that was preventing connection to one of the Adnaco PCIe expansion boards in the IO Chassis , see here: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=80035
The new frontend is running without issue, connected to all four Adnacos.
The new frontend model is Supermicro X11SRL-F, the same as the old frontend.
Tue Oct 01 08:09:24 2024 INFO: Fill completed in 9min 20secs
VACSTAT issued alarms for a BSC vacuum glitch at 02:13 this morning. Attachment shows main VACSTAT MEDM, the latch plots for BSC3, and a 24 hour trend of BSC2 and BSC3.
The glitch is a square wave, 3 second wide. Vacuum goes up from 5.3e-08 to 7.5e-08 Torr for these 3 seconds. Neighbouring BSC2 shows no glitch in the trend.
Looks like a PT132_MOD2 sensor glitch.
vacstat_ioc.service was restarted on cdsioc0 at 09:04 to clear the latched event.
This is the second false-positive VACSTAT PT132 glitch detected.
02:13 Tue 01 Oct 2024 |
02:21 Sun 01 Sep 2024 |
The FSS glitching isn't seen in this lock that ended a 11 hour lock, but our old friend the DARM wiggle reared its head again. (Attachment 1)
Useism came up fast, so many of the signals are moving a bit more, but DARM sees a 400ms ringup that I don't see other places. Again, the FSS didn't seem to be glitching at this time.
TITLE: 10/01 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive maintenance
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: USEISM
Wind: 2mph Gusts, 1mph 3min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.73 μm/s
QUICK SUMMARY: The IFO had just lost lock at low noise coil drivers as I arrived. I put it into IDLE and maintenance is already starting. Useism has grown to over 90th percentile, perhaps the cause of the lock loss, but I still need to look into this.
Workstations anhd wall displays were updated and rebooted. This was an OS packages update. Conda packages were not updated.
TITLE: 10/01 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 137Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: Observing at 140Mpc and have been locked for 7 hours. Range is still low unfortunately but it's been a quiet evening, but we did get another superevent!
LOG:
23:00 Observing and have been Locked for 1 hour
23:47 Superevent S240930du
We went out of Observing at 04:53:56, and then lost lock four seconds later at 04:54:01.
It looks like the reason for dropping observing at 04:53:56 UTC (four seconds before lockloss) was due to the SQZ PMC PZT exceeding its voltage limit, so it unlocked and Guardian attempted to bring things back up. This has happened several times before, where Guardian is usually successful in bringing things back and H1 returns to observing within minutes, so I'm not convinced this is the cause of the lockloss.
However, when looking at Guardian logs around this time, I noticed one of the first things that could indicate a cause for a lockloss was from the IMC_LOCK Guardian, where at 04:54:00 UTC it reported "1st loop is saturated" and opened the ISS second loop. While this process was happening over the course of several milliseconds, the lockloss occurred. Indeed, it seems the drive to the ISS AOM and the secondloop output suddenly dropped right before the first loop opened, but I don't notice a significant change in the diffracted power at this time (see attached screenshot). Unsure as of yet why this would've happened to cause this lockloss.
Other than the ISS, I don't notice any other obvious cause for this lockloss.
After comparing Ryan's channels to darm, still not sure whether this lockloss was caused by something in the PSL or not, see attached
There is a bump disturbance in the H1 strain spectrum that is degrading the noise in the vicinity of the Crab pulsar. Attached are zoomed-in ASDs from recent dates (drawn from this CW figure of merit site). That band was pretty stable in O4a, but has been unstable in O4b. I can see hints of the bump as early as the end of March 2024 (see calendar navigation to sample dates). I have poked around in Fscan plots and tried the handy-dandy Carleton coherence mining tool, but saw no obvious culprit PEM channels. Is there a new motor running somewhere (or an old motor with a drifting frequency)? Attachments: Sample ASD snippets from August 1, September 1 & 2, along with a snippet from the end of O4a on January 16. The red curve is from one day of averaging, and the black curve is the run-average up to that date. The vertical bars show the approximate band of the Crab pulsar during the O4 run (taking into account Doppler modulations and long-term spin down).
Correction: The graph labeled August 1 really applies to July 12, the last day of data before the long shutdown. Attached is what I should have posted instead (same curves with the correct label). Thanks to Sheila for spotting the error.
Edit : Neil knows how to make links, now. Tony and Camilla were instrumental in this revelation.
This is an update to my previous post (Neil’s previous post on this topic) about looking for possible explinations why similar seismic wave velocities on-site may or may not knock us out of lock.
The same channels are used in addition to:
SEI- * ARM_GND_BLRMS_30M_100M
SEI- * ARM_GND_BLRMS_100M_300M
Where * is C, D, X, or Y.
I have looked at all earthquake events in O4b, and only ones which knocked us out of lock. This is to simplify the pattern search, for now. Here are the results.
Total events : 29
Events with ISI y-motion dominant : 11 (30-100 Hz) : 8 (100-300 Hz)
Events with ISI x-motion dominant : 3 (30-100 Hz) : 0 (100-300 Hz)
Events with ISI z-motion dominant : 9 (30-100 Hz) : 19 (100-300 Hz)
Events with ISI xy-motion dominant : 1 : 0 (Both axes are similar in amplitude.)
Events with ISI yz-motion dominant : 0 : 1
Events with ISI xz-motion dominant : 0 : 1
Events with IS xyz-motion dominant : 5 : 0
Total SEI- * ARM recorded events : 8
CARM dominant events : 7 : 8
C/XARM dominant events : 1 : 0
Conclusion is that in the 30-100 Hz band, it is equally likely to have either z- or y-axis motion be dominant. In the 100-300 Hz band, the ratio is about 1:2 for z and y motion being dominant during lockloss.
Clearly, common modes are a common (^_^) cause of lockloss.
Note that velocity amplitudes should be explored more.
NOTE : All units are in mHz, not Hz.
Tue01Oct2024
LOC TIME HOSTNAME MODEL/REBOOT
08:23:11 h1iscey ***REBOOT***
08:24:51 h1iscey h1iopiscey
08:25:04 h1iscey h1pemey
08:25:17 h1iscey h1iscey
08:25:30 h1iscey h1caley
08:25:43 h1iscey h1alsey