REMOVE SEARCH FILTER
SEARCH AGAIN
Search criteria
Section: H1
Task: PEM
I made a new medm screen for the dust monitors so there's a single screen where you can see all of the particle counts for the site, and you can get to all useful dust monitor screens through this screen. I also added some shell command buttons to launch scopes of the individual DMs. Its located under PEM on SITEMAP.
Closes FAMIS 27648, last checked in alog 88314
Trends look fine!
J. Kissel Post Dec 4th power outage, we've have an EPIC week of windstorms that have inhibited recovery effort, which has delayed upgrade progress. The summary pages (on their 24 hour cadence) and the OPS logs / environment summary don't really convey this well, so here's a citable link to show how bad last Friday (12/05), Monday (12/08), and Wednesday (12/10) were in terms of wind. Given the normal work weekend, it means that we really haven't had a conducive environment to recover from even a normal lockloss, let alone a 2-hour site-wide power outage. The attached screenshot is of the MAX minute trends (NOT the MEAN, to convey how bad it was) of wind speed at each station in UTC time. The 16:00 UTC hour mark is 08:00 PST -- the rough start of the human work day, so the vertical grid is marking the work days. The arrow (and period where there's red-dashed 0 MPH no data) shows the 12/04 power outage. The horizontal bar shows the weekend when we humans were trying to recover ourselves and not the IFO.
Oh right -- and also on Monday, even though the wind wasn't *that* bad, the Earth was mad from the after shocks of 7.0 mag Alaskan EQ, and there were end-station Software Watchdog trips related to it that -- because of an oversight in watchdog calibration -- scared everyone into thinking we should "stand down until we we figure out if this was because the hardware upgrades or power outage." See LHO:88399 and LHO:88415. So, Monday was a wash for environmental reasons too.
I restarted all the dust monitor IOCs, they all came back nicely. I then reset the alarm levels using the 'check_dust_monitors_are_working' script.
WP12901. Daniel, Jeff, Oli, Fil, Marc, Jonathan, EJ, Tony, Dave:
Summary:
Yesterday I replaced h1iscex's 18bit-DAC with a 20bit-DAC as part of the project to eliminate all 18bit-DACs from production. There is currently no indication that the 20bit-DAC is driving the AI chassis with any voltage, investigation is ongoing.
Details:
The 18/20 bit DACs use the same interface card, so for the upgrade I was able to pull the IO Chassis out half way with cables still attached to access the PCIe bus. I removed the old 18bit-DAC and replaced it with on of the 20bit-DACs from h1susex upgrade
| old 18bit-DAC (removed) | 110425-03 |
| new 20bit-DAC (installed) | 190219-10 |
I do not know if the 20bit-DAC from h1susex was the one driving the PI or the one which had been idle since the LIGO-DAC went in mid 2024.
The original interface card and ribbon cable were reused. No field cabling was disconnected, no AI chassis were powered down.
Upgrade was done between 12:00 and 13:00 Tue 02dec2025.
Later that afternoon Tony found that the PCAL readbacks were not responsive. Tony and I went to EX around 17:00 to verify the 20bit-DAC was installed and connected correctly, it was.
This DAC has a special AI Chassis, D1101785 "aLIGO 18 Bit AI Chassis" which has handy BNC pickoffs for all 8 channels. We put a scope on the two channels being driven by h1calex (6,7) and could see no signal.
Also the last channel on the AI is loop-backed to the first ADC AA chassis with a DB9 cable, and the IOP is configured by SDF to drive the duotone from DAC0-chan7 back to ADC0-chan30. No signal is seen there either since the upgrade.
DCC docs for "18bit AI Chassis"
D1101785 [front section]
D1200316 [whole chassis]
The "AI WD" green LED on the rear panel is ON when the serial cable from the DAC Interface cable is connected and OFF when the cable is disconnected, suggesting the DAC is driving the WD line correctly at least.
Model changes:
h1iopiscex (Dave) updated to new DAC configuration
h1calex, h1pemex (Jeff, Oli) updated to new DAC configuration
All models on this front end, including h1iscex and h1alsex, were built and installed with RCG-5.5.2
Tagging CAL and PEM because this is "their" DAC.
We've done the typically thing with a DAC upgrade:
- the DC calibration of the DAC has changed from (roughly) 20 / 2^18 [Vpp_differential / ct] to (roughly) 20 / 2^20 [Vpp_differential / ct]
- Such that upstream control systems "don't have to be retuned" we apply a digital factor of (exactly) 4.0x to all output signals for each DAC channel
As such, for example, the calibration of the PCAL actuation / excitation chain is likely now slightly different, and should be remeasured.
h1pemex was upgraded from an 18-bit to a 20-bit DAC today, so we needed to make sure we had a calibration correction. The new filter banks that Jeff had put in (88321) for the calibration correction are called H1:PEMEX_EX_DACOUTF_1, H1:PEMEX_EX_DACOUTF_2, H1:PEMEX_EX_DACOUTF_3, and H1:PEMEX_EX_DACOUTF_4. I installed a filter called 20BitDAC that was a gain(4) in FM10 of each of these filter banks, loaded them in, and turned them on along with the input/output/gain of the filter banks. I've accepted these changes in sdf safe
Late update, but I put the channel names wrong. The new channels are called H1:PEM-EX_DACOUTF_{1,2,3,4}.
I've installed a Welch WOB-L 2546B-01 at each end station to replace the Gast 523 pumps that we have been using. I've set the pressure to -19inHg and checked the flow at the dust monitor on the floor was 2.8L/min. The new pumps are much quieter, ~84dB vs ~67dB, but at a lower frequency. I'll post more photos of the setup and more details tomorrow after we see how they run over night.
J. Kissel, D. Sigg, D. Barker ECR E2100485 ECR E2200401 WP 12901 Continuing on the deprecation of 18-bit DAC path (ECR E2100485), we upgraded the h1iscex's DAC0 card today from an 18- to a 20-bit DAC. One of the front-end models that use that DAC is the h1pemex model. Here's the before vs. after for the models. While there, I found and left the new ADC card from ECR E2200401's PEM sensor array expansion. I'd thought it was installed solely for characterizing the 32CH LIGO DAC, but it had only been temporarily used for that. It should be there! And also, the electric field meter ADC should also remain there. If the PEM team wants to account for the factor of 4x gain change in the DAC calibration, I installed new DACOUTF filters upstream of the GDS filters that are used to drive the DACs. The model has been committed to /opt/rtcds/userapps/release/pem/h1/models/ h1pemex.mdl : r28026 --> r34059
For FAMIS #27646: All fan trends look nice and flat!
Closes FAMIS37258, last checked in alog86733
Everything's looking as expected, no followup investigations needed.
Last night I ran a longer version of the full magnetic injection suite starting at about 03:10 UTC, and it finished at 07:33 UTC. This used the same parameters as last week's run, except I doubled the injection time to 5 minutes each.
Exact times and injection parameters are in the usual log directory for the magnetic injections: /ligo/www/www/exports/pem/WeeklyMagneticInjection/logs/1448075483.txt
The last batch of injections (all 7 of the ones at EY E-bay) mistakenly overlapped with a scheduled SQZ script, so last night I re-ran these between 05:29 and 06:06 UTC.
Analysis results and plots can be found here: https://ldas-jobs.ligo.caltech.edu/~ryan.short/pem/Weekly_Mag_Inj/H1_Injection_Results/O4/20251125_1448075483/
At some point within the last year, both the X and Y-axis accelerometers (PEM-CS_ACC_LVEAFLOOR_HAM1_X and Y) were kocked off (pictured here) the small metal cube they were mounted to. I reattached them as of Nov 11, 2025. I believe the capacitor position sensor box was pushed back into the accelerometers which knocked them off. Image 3 shows where the CPS box is in relation to HAM1 (circled in red) and image 4 shows where it is in relation to the accelerometers.
Additionally, two other accelerometers have been remounted with epoxy in August: PEM-CS_ACC_BEAMTUBE_SRTUBE_X (remounted August 5, 2025) and PEM-CS_ACC_BEAMTUBE_YMAN_X (remounted August 12, 2025). Both had fallen off the beamtube and had been hanging in the air for an unknown amount of time until someone noticed.
The HAM 1 floor accelerometers appear to have been moved on or around April 18, 2025 during the HAM1 work during the break between O4c.1 and O4c.2. The coherence between the two signals dramatically changes character after that date. I attach some plots of the coherence between accelerometers the day after you repositioned them, as well as data from 0000-0100 UTC on a few example days. I'm not sure that it was moved exactly on April 18, but the coherence data begins to differ from the reference on that date.
Since I saw H1 was locked, I started the full magnetic injection suite (49 injections total over the 7 coils across site) at 05:40 UTC using the PEM_MAG_INJ Guardian. This dropped H1 out of observing, but it should return to observing once the injections finish. The suite will finish in roughly 2 hours or if the IFO loses lock, whichever comes first. I'll check things later tonight in case this poses any issues.
Injection suite finished at 08:00 UTC and H1 returned to observing.
I've ran a few different huddle test with the new TemTop dust monitors against 3 different MetOne GT521s. The first tests I ran in the Optics lab and Diode room showed some discrepancies between the spike times but that could be due to a difference in sample times vs the two dust monitors. The PMS21 and GT521s both sample for 60 seconds then hold for 300s, but the PMD331 does not have a built in hold time so it samples continuously or manually. The PMS21 usually reads an order of magnitude or two lower than the other two which makes me wonder if it needs a scale factor, but it also sees these big spikes occasionally that the others don't see which is confusing. The flow rate is listed as being 0.1 CFM on the PMS21, but the GT521s and the PMD331 are also listed as having flow rates of 0.1 CFM. CFM = Cubic feet / minute, its also = 2.83 L/min which is what I read when running the flow test on all of the DMs. *Also the times do not account for daylight savings, so each y-axis timestamp is actually an hour behind the actual PST*.
Test 1 - Optics Lab:
I tested the TemTops one at a time in the Optics lab, the results of the PMD331 and the results of the PMS21.
Test 2 - Diode Room:
I tested the TemTops at the same time in the Diode room, I started off with only the PMS21 then I added the PMD331. For only the PMS21 we saw the these counts, for only the PMD331 we saw these counts, and for both of them we saw these counts,all against a MetOne GT521s.
Test 3 - Control Room:
I tested three dust monitors at the same time in the control room ( I grabbed a spare pumped GT521s from the storage racks by the OSB receiving, it's our last properly working spare). I did a day of samples with holds enabled and a day of continuous sampling. When the dust monitors were sampling at slightly different intervals we saw the peaks offset from each other such as at 11-18 ~07:00 PST at the right of the plot. During the continuous testing we can see the peaks from everyone coming into the control room for the end of O4 celebration, I'm not sure why there's some time between the peaks, zooming in on this plot to cut out the large peaks from said celebration we can see the PMD331 and the MetOne GT521s following each other pretty closely, but the PMS21 wasn't really reading much, there are small bumps around where the peaks from the other DMs are. Adding a scale factor of 10 to the PMS21 counts yield a better looking plot, playing with the scale factor till the PMS21 counts looked more in-line with the other DMs I got to a scale factor of 40 to get this plot.
Famis 27573 Vibration Sensors To Gauge Health Of HVAC Fans Site
H0:VAC-MR_FAN6_170_1_ACC_INCHSEC seem to be going up in that few days especially the last 24 hours.
This is also the case for H0:VAC-MY_FAN2_270_2_ACC_INCHSEC in the last day saw a fairly large increase in noise.
